Emotion detection using artificial intelligence with spatiotemporal data obtained from body language

This study provides a comprehensive evaluation of the effectiveness of spatiotemporal data and multidimensional approaches in recognizing emotions through body movements. By utilizing a kinematic raw dataset and the video-based DEMOS dataset, it compares the performance of various methods for classifying fundamental emotions, including anger, disgust, fear, happiness, neutrality, sadness, and surprise. While methods based on facial expressions and voice dominate the literature, this study highlights the potential of body movement-based emotion recognition, particularly in scenarios where facial expressions are insufficient. In the analysis of kinematic data, raw skeletal position information was assessed both as unprocessed data and after feature extraction. The study tested a range of machine learning algorithms, including K-nearest Neighbors, Random Forest, CatBoost, and XGBoost, alongside deep learning models such as RegNetY, MobileNetV3, LSTM, and GRU. For the seven emotion classes, the highest accuracy rate exceeded 99% across different windowing sizes, demonstrating that emotion recognition from raw kinematic signals is highly feasible with remarkable precision. Experiments on the DEMOS video dataset tested spatiotemporal data for six emotion classes using deep learning methods (such as SlowFast-R50, X3D-Medium, ResNet-3D-18, and Attentive3D-CNN-LSTM).With video data captured from all angles, the highest balanced accuracy rate for the six emotion classes reached 60% across all test data. The results show that raw kinematic data, with its high accuracy, can be effectively used in multi-class emotion classification. Additionally, combining skeleton-based video data with its contextual richness in multimodal approaches holds significant promise for improving emotion recognition. The study highlights broad application potential in fields such as human-machine interaction, security, healthcare, and education. Moreover, it emphasizes that techniques like signal processing, feature extraction, data augmentation, and transfer learning could substantially enhance the efficiency of emotion recognition processes. This study compares kinematic and video-based datasets, presenting an innovative framework for the development of diverse data systems in emotion recognition. It establishes a solid foundation for advancing emotion recognition technologies and makes a valuable contribution to the literature by offering both methodological and practical recommendations for future research.