Deep reinforcement learning to optimize task performance in human-robot co-manipulation

We propose a two-layer machine learning (ML) approach, which utilizes an artificial neural network (ANN) model as a precursor for a deep reinforcement learning (DRL) model, to optimize task performance during human-robot co-manipulation of heavy objects. In the first layer, the ANN model estimates the human intention to accelerate or decelerate the object (which precedes the actual acceleration or deceleration of the object due to its large inertia). This probabilistic estimation is then used to calculate the gain of an adaptive admittance controller, which alters the robot's contribution to the task. In the second layer, the DRL model fine-tunes this gain and optimizes the task performance by minimizing the jerk in movement and the physical effort made by human. Since online training of a DRL model for a physical human-robot interaction (pHRI) task is highly time-consuming and can potentially be dangerous for the human due to abrupt changes in controller gain, a data-driven human force model was developed by a conditional variational auto-encoder (C-VAE) for offline training of the DRL model via simulations. For this purpose, experimental data was collected from six subjects under 3 different fixed gains of the admittance controller (minimum, nominal, and maximum) to train and validate the DRL model. The adaptive admittance gain profiles generated by the ANN model alone and the proposed two-layer approach (ANN + DRL) were compared through co-manipulation simulations. The results show that the gain profile obtained by the two-layer approach leads to a decrease in human effort and jerk compared to the initial profile provided by the ANN model.