D. Reinforcement Learning Controller (Demo)

Objective

The objectives of this laboratory experiment are as follows:

1. Train a controller that balances the pendulum using Reinforcement Learning.

2. Implement the balance controller on the Quanser Rotary Pendulum system and evaluate its performance.

Equipment

• Rotary pendulum (Quanser QUBE-Servo 3 Rotary inverted pendulum module)

• Rotary servo base (Quanser QUBE-Servo 3)

• MATLAB and SIMULINK

QUBE-Servo 3 Rotary Inverted Pendulum Model

A picture of the QUBE-Servo 3 with rotary pendulum module is shown in Fig 1. The numbered components in Fig. 1 are listed in Table 1 along with the numerical values of the system parameters in Table 2.

Figure 1: QUBE-Servo 3 base with Rotary Pendulum module

Table 1: QUBE-Servo 3 components (Figure 1)

ID#	Component
1	Rotary servo
2	Rotary arm housing
3	Pendulum encoder
4	Rotary arm
5	Pendulum link

Table 2: QUBE-Servo 3 main parameters

Symbol	Description	Value	Unit
	Mass of pendulum	0.024	$\mathrm{kg}$​
​	Total length of pendulum	0.129	$\rm{m}$
​	Pendulum moment of inertia about center of mass	3.3282 · 10-5	$\rm{kg \cdot m^2}$
	Pendulum viscous damping coeffi­cient as seen at the pivot axis	5 · 10-5	$\rm{N\cdot m\cdot s/rad}$
​	Mass of rotary arm	0.095	$\rm{kg}$​
	Rotary arm length from pivot to tip	0.085	$\rm{m}$​
​	Rotary arm moment of inertia about its center of mass	5.7198 · 10-5	$\rm{kg \cdot m^2}$​
​	Rotary arm viscous damping coeffi­cient as seen at the pivot axis	0.001	$\rm{N\cdot m\cdot s/rad}$
$R_m$	Motor armature resistance	8.4	$\Omega$
$k_t$	Current-torque constant	0.0422	$\text{N}\cdot\text{m} / \text{A}$
$k_m$	Back-emf constant	0.0422	$\text{V}\cdot\text{s} / \text{rad}$

Reinforcement Learning

Fundamentals

Although we have found a sufficient model for the Inverted Pendulum system in Part A, there are often systems where the plant dynamics are difficult or impossible to model. In this case, we look to model-free control.

Reinforcement learning algorithms, which include model-free algorithms, seek to train an agent. The agent receives observations of the plant system in addition to feedback from a reward function, and outputs a corresponding action according to its policy. The agent may be trained on the physical system in a series of discrete training episodes. Following each episode, the agent updates its policy according to the specific reinforcement learning algorithm in use. A summary of the reinforcement learning training loop is shown in Fig. 2.

Figure 2. Reinforcement learning training loop (Mathworks 2025)

Following training, the agent may be deployed on the system to ideally achieve the desired behavior. The particular reinforcement learning algorithm used in this lab is the deep deterministic policy gradient (DDPG) algorithm, a model-free actor-critic algorithm. For more information about the DDPG algorithm in particular, please visit the https://www.mathworks.com/help/reinforcement-learning/ug/ddpg-agents.html.

Reinforcement Learning for the Inverted Pendulum System

To implement the DDPG algorithm, we need a reward function, which evaluates the observed performance of the system. A good reward function promotes behavior we desire in the system, and penalizes behavior we want to eliminate. In this case, we seek to command the system to $\theta =0,\,\alpha=0,\,\dot{\theta}=0,\,\dot{\alpha} =0.$ Refer to Part A Modeling for the definition of variables. Additionally, we especially want to avoid the servo angle $\theta$ exceeding the physical limit of the system at $\theta_{\text{max, actual}} = 90\degree$, with a margin of error such that $\theta_{\text{max}} = 60\degree$ , the pendulum angle $\alpha$ exceeding the balance control limit of $\alpha_{\text{max}}=\epsilon=12\degree$, and the input voltage limit of $V_{\text{max}}=5\,\text{V}$ . Thus, we use the following quadratic reward function

$$Reward = -(q_{11}\theta^2+q_{22}\alpha^2+q_{33}\dot{\theta}^2+q_{44}\dot{\alpha}^2+r_{11}u^2)+B(|\theta|>\theta_{\text{max}} \,OR\,|\alpha|>\alpha_{\text{max}}\,OR\,|u|>u_{\text{max}}),\qquad \qquad\quad \tag{31}$$

where $q_{ii}.\,r_{ii}, B$ are weights. This function covers all of the desired behavior we seek from the system. By default, the weights are $q_{33}=0<r_{11}=0.1<q_{44}=1<q_{11}=10<q_{22}=20,$ and $B=-100,$ reflecting the relative importance of each state.

For a model-free application, an engineer would typically train the reinforcement learning model on the physical plant system or a digital twin of the plant system. However, doing this during a 2.5 hour lab session is impractical, as this would require repeatedly raising the pendulum for 1000 training episodes. Consequently, we train our reinforcement learning agent in simulation on the state space model we developed in part A.

A. Training the Reinforcement Learning Controller

764KB

Lab4_Group.zip

archive

Open

1. Download the Lab4_Group.zip file to the lab PC and extract the folder contents.

2. Open the rotpen_group.mlx live script and run the Initialization section. This section generates all necessary parameters for the reinforcement learning training. Ensure that the drop down menu is configured to 'train new agent.'

3. Run the Observation and Action Signals section. This section creates the learning environment for the reinforcement learning agent, configures the observation signal read by the agent, and configures the action signal generated by the agent.

4. Run the Create DDPG Agent section. This section creates and configures the critic and actor neural networks, and compiles these networks into a single agent.

5. Run the Train/Load Agent section. This section begins the training session. By default, the training session is configured to contain 1000 episodes with a random starting pendulum angle between -20 and 20 degrees.

6. The s_rotpen_train_rl.slx Simulink diagram should automatically open, in addition to a Reinforcement Learning Training Monitor window. Open the 'alpha' and 'theta' scopes and configure your monitor so you can clearly see all three windows, as in Fig. 3. You should now be able to watch as the reinforcement learning agent trains on your state space model.

7. Wait for the training to complete. This should take 20-30 minutes. Ensure that the main MATLAB window is open and visible to avoid slowdown due to CPU allocation. On training completion, your trained agent will be saved to the current directory as agent_student.mat.

Figure 3. Screen configuration for reinforcement learning training.

B. Implementing the Reinforcement Learning Controller

1. Proceed to the Implement Reinforcement Learning on Hardware section of the rotpen_group.mlx script.

2. Ensure the 'doPolicy' drop down box is set to 'true'.

3. Run the section. The q_rotpen_rl_student.slx Simulink diagram should open automatically.

4. To build the model, click the down arrow on Monitor & Tune under the Hardware tab and then click Build for monitoring . This generates the controller code.

5. Press Connect button under Monitor & Tune and Press Start .

6. Once it is running, manually bring up the pendulum to its upright vertical position. You may need to try this a few times.

7. Observe the behavior of your controller. It is possible that your reinforcement learning controller is unable to balance the pendulum, or, if it is, the control is not very robust. This is OK! These are the natural consequences of Reinforcement Learning based control.

Questions for Report

1. Briefly summarize the RL balance controller experiment and your observations. How did the RL controller behave compared to the model-based pole placement controller?