A new approach using deep learning methodologies from human activity recognition to Robot Grasping

The research goal of this work is to develop an advanced deep learning model to analyse automatically human motion on videos and to present the prototype of a basic robotic imitation system that mimics the human movement supported by deep learning. For this purpose, we propose a hybrid deep model for human activity recognition, robotic simulator infrastructure for human imitation, and simple, intelligent, extensible and pluggable video analytic framework for integration. First, we present a new hybrid deep learning model for human activity recognition in videos. We proposed a new 3-stream hybrid deep model with data fusion of 3D-CNNs fed by dense optical flow and LSTM fed by auxiliary information. We used SVM as a classifier. We generated 2 different datasets, namely the magnetic wall chess board video dataset (MCDS), and standard chess board video dataset (CDS). They consist of microvideos with duration of 5-6 second that contain meaningful movements by the chess player. We experimented hybrid deep model with two new datasets, the experimental results show remarkable performance compared to the state-of-the-art studies. The proposed hybrid deep model can be used in complex motion recognition tasks, thanks to its ability to fuse information with different characteristics such as motion, image, sound and text. Second, we developed a robotic simulation infrastructure for chess-playing delta robot. We used V-REP virtual robot experiment platform by Coppelia Robotics. Generated robotic simulator can operate both standalone and by being controlled by an external system. Finally, we designed a simple video analytic framework for human motion imitation system and used our hybrid deep learning model in this framework. We tested end to end system with a specific scenario in offline mode. In this way, we have developed an artificial intelligence supported human imitation system prototype to be used in the specific problem of imitating chess player by robot simulator. The robot simulator in the proposed system can imitate the chess player with motion primitive approach. As a result, we achieved an AI-powered, intelligent, extensible, and pluggable human imitation system prototype.