Deep learning-based emotion recognition on video

Automatic Emotion Recognition (AER) systems for human-machine interaction are foregrounded as a topic of study that is growing in popularity. In this thesis, a new model based on deep learning that uses visual and auditory features together has been proposed for AER systems. With the proposed model, auditory and visual features are extracted separately from the video sequences and combined in a three-dimensional feature capsule. These obtained three-dimensional feature capsules were also classified by using a three-dimensional deep learning based classifier. In this study an originally designed 3DCNN-LSTM network model was proposed by combining an attention layer added 3D Convolutional Neural Network with Long Short-Term Memory end-to-end fashion. In the proposed study Spectrogram, Mel-Frequency Cepstral Coefficient (MFCC) feature maps, Cochlegram and windowed fractal feature maps were used to convert speech signals into the images in the auditory feature extraction stage. A two-stage method was applied in the extraction of visual features. Firstly, video sequence was summarized by choosing face frames detecting for emotion recognition with an algorithm which is also named as video summary and based on key frame selection. Then geometric features were extracted by finding the coordinates of landmark points on the face from the obtained key fames in this stage and these features were converted into feature map images. All the feature images consisting of auditory and visual features in th training and testing stages were classified with three-dimensional neural network model designed by being combined in a feature capsule. The proposed method was tested by using the CREMA-D, RAVDESS, SAVEE and RML datasets prepared in video format and the classification performances were discussed. The obtained results in this study were compared with the recent studies and it was seen that the proposed method showed a much better performance than the other studies. Keywords: Automatic Emotion Recognition, Deep Learning, 3D Convolutional Neural Network, Long Short-Term Memory, Fractal, Mel-Frequency Cepstral Coefficient