Multimodal sentiment analysis using deep learning methods

Multimodal sentiment analysis enables a more comprehensive examination of emotions by jointly evaluating different data modalities such as text, audio, and visual cues. Linguistic structures and word choices in text, prosodic features such as intonation and pitch in audio, and facial expressions in visual data provide complementary signals for understanding emotional states. This approach has been widely applied in areas including social media analysis, healthcare, education, and marketing. In this study, a multimodal emotion recognition model was developed by jointly utilizing text and audio modalities. Experiments were conducted on the MELD (Multimodal EmotionLines Dataset) under three labeling scenarios: 3 emotions (negative, neutral, positive), 5 emotions (anger, sadness, joy, neutral, surprise), and 7 emotions (anger, sadness, joy, neutral, surprise, fear, disgust). For the text modality, contextual representations were obtained using a pretrained RoBERTa model, while HuBERT was used for audio representation learning through a self-supervised approach. The extracted HuBERT features were classified using a lightweight MLP-based architecture. Text and audio modalities were integrated using a late fusion strategy, where class probabilities from separately trained models were combined at the decision level. Model performance was evaluated using Weighted F1 alongside accuracy and additional analyses such as ROC curves and confusion matrices. Results indicate that the late fusion model provides more consistent performance compared to single-modality approaches, achieving WF1 scores of 72, 66, and 62 for the 3-, 5-, and 7-class scenarios, respectively.