Development of deep learning based multimodal sentiment analysis methods

The wide use of social media apps by the public has allowed people to provide rapid feedback on a situation, event, service or product. In sentiment analysis, it is possible to determine people's feelings and thoughts about a subject or product from social media posts. Basic sentiment analysis approaches are generally performed on a single modality. As a result of the widespread use of the Internet and smart devices and the diversity of social media applications, the data produced daily can be in text, image, video, and audio. Multimodal sentiment analysis is the process of revealing the sentiment of user post by analyzing different modality as a whole. In this thesis, multimodal sentiment analysis was conducted on the benchmark datasets collected from social media. Two approaches are considered for multimodal sentiment analysis. The first is based on the use of multimodal deep learning models. In this approach, a novel hybrid deep learning model has been developed that strategically uses different text representations (Character-level, Word2vec, FastText) together with different deep learning methods (LSTM, GRU, BiLSTM, CNN). The proposed model for this approach extracts features from different representations with different deep learning methods, and classifies texts in terms of sentiment by combining features. The second approach is a method based on sentiment analysis on multimodal datasets. An ensemble learning model based on soft voting is proposed that takes advantage of the effective performance of different classifiers on different modality. Sentiment analysis results of the proposed methods have been verified with experimental studies.