Teleoperation of an industrial robot arm by analyzing human arm depth image sequences

In this study, human operator's arm movements synchronized to 6 DOF Stäubli Rx160 industrial robot arm by using Microsoft Kinect ToF camera. Kinect can provide depth images at 30 Hz rate in 640x480 resolution. By using Kinect SDK that is provided by Microsoft, depth images are analyzed and operator's depth map is obtained. Once depth map is obtained operator is modelled with a stick figure with 20 joints. Each joints' three dimensional position data can be obtained from the depth map using Kinect SDK. Vectors that represents the bones between joints are calculated using joint positions on operator's right arm shoulder, elbow and wrist joints. These vectors are then used for calculation of operator's arm angles. Operator's workspace extended to Stäubli Rx160's workspace by using arm angles and Stäubli Rx160' dimensions. Human arm kinematic model and its forward kinematic equations are calculated for checking the validity of the calculated arm angles. Calculated angles are filtered using human arm forward kinematic equations vii to eliminate angles that does not match human arm's kinematic model. After obtaining angles that match human arm kinematic model, three different teleoperation methods applied and tested. These methods are used to control Stäubli Rx160's end position with 3 DOF. Stäubli Rx160's last three joints represents orientation of its wrist joint. Wrist orientation is neglected in this study, thus last three joints of Stäubli Rx160 is setted to a position where end effector is in line with the robot arm's orientation. For the first method, direct angle calculation is used. In order to send angles directly, human arm joint anlges are mapped to Stäubli Rx160's joint angles. Inverse kinematics calculation method is used for the second method. Wrist position of the operator is calculated from the extended workspace parameters and used as robot's end effector position. Robot's joint angles are calculated from end effector position. Inverse kinematics calculations resulted in different solution sets where all sets lead to the given end effector position. These solution sets are filtered to obtain elbow-down operation mode which resembles operator pose. For the third teleoperation method, an artificial neural network is trained using data sets obtained from inverse kinematics solution method. Two different neural networks are trained using Matlab's Neural Network Toolbox. Accuracy of the trained networks, which are trained with different number of data points, are compared and discussed. Calculated angles are sent to Stäubli Rx160's emulator using TCP/IP protocol. A VAL3 program on the emulator accepted the connection and read data from the incoming stream. Then this program converts incoming data stream back to robot joint angles and sends angles to corresponding joints motor driver. Developed teleoperation methods are compared against each other by means of accura cy and calculation time performance. Results are discussed and recommendations for future work are presented at the conclusion.