Application of a voting-based ensemble method for recognizing seven basic emotions in real-time webcam video images

Automatic recognition of human emotions based on facial expressions is a challenging task with significant implications in various fields, including human-computer interaction, healthcare, and affective computing. In recent times, modern deep learning techniques, particularly Convolutional Neural Networks (CNNs), have exhibited promising results in the domain of facial emotion recognition. The study presents comprehensive research to reveal the most effective method for recognizing seven basic emotions from 2D facial images on real-time video or real-time webcam. The research investigates and compares different methods, including Data Augmentation methods, CNN models, KNN models, and a hybrid CNN-KNN approach. The proposed hybrid CNN-KNN method involves leveraging the rich feature representations learned by a pre-trained CNN model for emotion analysis. The pre-trained CNN extracts high-level features from facial images, which are then used as input to a KNN classifier for emotion classification. The thesis evaluates the hybrid CNN-KNN approach against traditional standalone CNN models and KNN models to assess its performance and effectiveness on real-time video. In addition to the hybrid CNN-KNN method, the thesis explores several other approaches to facial emotion recognition, including different CNN architectures, transfer learning using pre-trained models, data augmentation techniques, and ensemble methods. The aim is to thoroughly analyze and compare the performance of these methods and determine the optimal approach for accurate emotion recognition. The assessment is carried out using the FER2013 dataset, a well-established collection of labeled 2D facial images representing seven fundamental emotions. To comprehensively gauge the methods' effectiveness, a range of performance metrics including accuracy, precision, recall, F1 score, real-world experiments and computational efficiency are employed. This research's findings shed light on each approach's strengths and weaknesses and identify the most effective method for facial emotion recognition. The results will guide the development of emotion recognition systems in real-world applications, enabling more empathetic and context-aware human-computer interactions. Our main motivation is investigating the effectiveness of ensemble methods in emotion recognition. Our side motivations are to develop FER systems and start a new DB to be beneficial to both Turkiye and the world, improving communication between humans and machines, making technology more sensitive to human emotion, supporting technology to understand humans and humans to understand technology. As a result of this research, Successfully attained a remarkable 95% accuracy on the amalgamated FER2013, CK+, and KDEF datasets, leveraging a comprehensive support base of 29,716 instances. Introduced a novel database, ATS_FER_DB_2023, achieving a commendable accuracy of 94% on merged new DB and FER2013. This database encompasses 86 images, featuring prominent Turkish celebrities. In the realm of real-time emotion recognition, a meticulous comparison of various methods within our environment revealed that the CNN-KNN algorithms, enhanced with Principal Component Analysis (PCA), emerged as highly effective in our specified system assembly.