Multimodal emotion recognition in video

This thesis proposes new methods to recognize emotions in video considering visual, aural, and textual modalities.In visual modality, we proposed a new facial expression recognition algorithm based on curve fitting method for frontal upright faces in still images. Proposed algorithm considers the shape of mouth region to recognize happy, sad and surprise emotions. According to our experiments, our method achieves 89% average accuracy. In addition, we proposed a skip frame based approach for video segmentation.In aural modality, we present an approach to emotion recognition of speech utterances that is based on ensembles of Support Vector Machine classifiers. In addition, we proposed a new approach for Voice Activity Detection in audio signal, and presented a new emotional dataset called Emotional Finding Nemo based on a popular animation film, Finding Nemo.In textual modality, we proposed an emotion classification method based on Vector Space Model (VSM). Experiments showed that VSM based emotion classification on short sentences can be as good as other well-known methods including Naïve Bayes, SVM, and ConceptNet on predicting emotional class of a given sentence.Finally, we use late fusion technique with a web-based interface for emotional browsing of TRECVID dataset, and we developed an emotion-aware video player to demonstrate the system performance.