Emotion classification with artificial neural networks from facial expressions and gestures

Classifying the right emotion from different data sources such as text, images, video, and speech has been an inspiring field for researchers from various disciplines. Automatic emotion detection from videos and photos is one of the challenging topics being studied using supervised and unsupervised machine learning methods. In this thesis, several preprocessing steps and a new deep learning architecture and emotion classification method from videos are presented. The face and body position information obtained from the videos using the OpenPose tool was converted into pose descriptors for use in the models, then the LSTM and Transformer models were trained with this data and their performances were compared. Then, the output of the CNN block fed with pose descriptors was used as input for the LSTM and Transformer models. Video Generation, Keyframe Selection, and Gaussian Mixture Center approaches were added as preprocessing steps to improve model accuracy and the experiments were repeated for combinations of these different approaches. After extensive experiments, the results were compared and the effects of the proposed two-layer classifier structure and preprocessing steps were observed. Results were also compared with other recent, high-accuracy methods using the same dataset. Experiments using two common datasets, FABO and CK+, showed that the CNN-Transformer structure with video generation approach for the FABO dataset outperforms the other models, with an accuracy of 99%. For both datasets, the proposed method in many versions achieved remarkable success, reaching an accuracy of over 90%.