Human activity recognition using deep learning

Human activity recognition has been very active research topic in computer vision area due to its wide range of applications, such as smart surveillance, robotics and human-computer interactions. Researches on activity recognition mainly focused on RGB videos and wearable sensors and its applications. With easy availability and and widespread use of RGB-D sensors, human activity recognition is becoming more popular. Activity recognition can be applied to many areas like detecting dangerous events or monitoring people living alone. For example, if a robot could watch and keep track of how often a person drinks water, it could prevent the dehydration of elderly by reminding them. Due to its unstructured and often visually confusing nature, recognition of daily activities becomes a much more difficult task. Activity recognition problem can be defined as classifying activities as robust as possible without depend on environment, camera positions, lighting conditions background changes and so on. This is very challenging task because of it may contain many parameters. Also nature of the activities changes dramatically person by person. Also problem scenario may contain many variation: we can have interacting activities, we can have very different camera locations or we can have different datasets etc. So, human activity recognition is still open problem and there is no prominent solution to solve all different scenarios. In this thesis, human activities were classified with various deep learning techniques using skeletal and RGB datasets. We aimed that to solve that problem without depend on just one type data. Recent years, deep learning based approaches has shown great success on many areas such as image classification, object recognition etc. They have great feature extraction capabilities if we compare to hand-crafted feature extraction techniques. So, we constructed from scratch deep neural network to get these meaningful features. The aim of processing skeletal data is to automatically recognize human activities with deep learning techniques using three-dimensional skeletal joint data from the RGB-D sensor. Depth sensors open up possibilities of dealing with the analysis of human actions based on 3D skeleton data has become popular recently, due to its robustness and view-invariant representation. Our methods uses the joint data directly and automatically acquires the features to be used in the classification, thus provides superiority to the methods which uses hand-crafted features. In our work, we used several datasets for human activity classification. One of them is NTU RGB+D dataset which is quite new and challenging compared to the datasets in the literature is used. Also, this dataset is very large and very suitable to implement deep learning which is data-hungry methods and it includes interacting activities. We have carried out several analyzing methods for classification of interacting activities. Also, we have seen that there is no enough work in the literature for classifying interacting activities. It is important how your model is good enough to classify real world data you get. So, we collected skeletal data using Microsoft Kinect v1 sensor from 4 different person for five different activities.We trained our model on MSR Daily Activity 3D dataset and tested our model on data we collected. Test results shows that our models very good at new person activities. With using 2D, 3D from scratch Convolutional Neural Networks(CNN) and Long-Short Term Memory(LSTM) Networks a performance analysis was performed for skeletal data. Also a novel technique was implemented for data augmentation and new synthetic data was created by using Generative Adversarial Network (GAN). By using this technique, accuracy was increased significantly. We used two different Long-Short Term Memory (LSTM) models for classification, standard LSTM and Bidirectional LSTM. Standard LSTM network have restrictions as the future input information cannot be reached from the current state. The basic idea of Bidirectional LSTM model is to connect two hidden layers of opposite directions to the same output. Results shows that Bidirectional LSTM has great capability to interpret relation of sequential data with each other. Several Convolutional Neural Network (CNN) based methods was implemented for activity recognition. Firstly, we developed two different from scratch 2D CNN for skeletal data. We analyzed how deepening network is effecting learning capability.In another experiment we used two-stream CNN network where model is fed another feature map which include motion information between consecutive frames as a supportive input. To get more meaningful spatio-temporal informations about activity, in another experiment we deployed 3D CNN model for this task. Also, we used Support Vector Machines (SVM) as a classifier after obtained features by 3D CNN. Thus, we wanted to analyze classification success between SVM and Multilayer Perceptron (MLP). Experiments shows that 3D CNN architecture developed from scratch gives a higher performance than other CNN approaches and it enables action representation properly. Success of the deep learning based solutions rely on size of the training data. For this reason, we used large in this thesis. However, we tried to augment data by using a novel technique called Generative Adversarial Network (GAN).Generative Adversarial Networks (GANs) offer a novel way to unlock additional information from a dataset by generating synthetic samples with the appearance of real data. By using GAN, 20000 new synthetic data were created and included to training set before classification. As a result of the experiments made, the technique applied by the 3D CNN and combined with GAN achieves the highest classification accuracy with by obtaining much more meaningful features compare to the LSTM Networks. In the second part of the thesis, we propose different deep learning techniques for activity recognition by using RGB data in videos. We carry out three different experiments which are 3D CNN architecture solely, hybrid model which deploy LSTM network for classification, and combination of optical flow with features from 3D CNN networks. Experiments shows that optical flow features are very useful to get motion features of activities and combining it with 3D CNN architectures helps to increase accuracy and allows more robustly classification. UCF101 and HMDB51 benchmark datasets were used for performance analysis and evaluate the proposed methods. These are very challenging datasets because of they consist of cluttered background and contains highly camera motions and also has unconstrained nature. Results shows that combining optical flow features with 3D CNN features makes better representation for motion understanding and more robust classification.