Developing walking skills of humanoid robots with deep reinforcement learning algorithms

Developing robust locomotion for humanoid robots is a challenging problem that has been researched for decades. Although various walking approaches have been proposed and walking performance has been significantly improved, it still falls short of expectations in stability. Low convergence and training efficiency for reinforcement learning approaches limit their applications. To overcome such limitations, an effective framework based on Robotis-OP2 humanoid robot combined with traditional trajectory generator controller and Deep Reinforcement Learning (DRL) is proposed in this thesis. This framework consists of the optimization of the gait trajectory parameters and the posture stabilization system. In the Webots simulator, gait parameters are optimized using the Dueling Double Deep Q Network (D3QN), one of the DRL algorithms. The hip strategy is adopted for the posture balancing system. Experimental studies are carried out in both simulation and real environment with the proposed framework and the Robotis-OP2 humanoid robot's own walking algorithm. Experimental results show that the robot performs the task of straight walking with the proposed framework more stable than the own algorithm of the robot. Later, within the scope of the thesis, two separate gait stabilization frameworks, consisting of a PID controller and a DRL controller, are proposed for the robot to walk stably on sloped surfaces. The DDPG (Deep Deterministic Policy Gradient) algorithm is preferred as the DRL controller. In the experimental studies performed with the PID controller, the posture of the robot was adjusted in real-time to ensure that the body pitch angle is at the desired reference value. With the DDPG controller, the sequential movement sequence of the robot's body pitch angle is learned in order to ensure the robot's balanced walking on inclined surfaces. Experimental results show that the DRL controller is more useful than the PID controller and provides a more stable gait.