Multimodal classifier for disaster response

Nowadays, social media data can be used to make a huge difference in making correct decisions in time-critical situations, such as the event of natural disasters. Social media content consists of messages, images, and videos. In some cases, understanding the damage caused by natural disasters only from text is not enough in terms of analysis, the effect of disaster is better understood using visual data. Text datasets from social media platforms are widely used by researchers, and a limited number of studies have focused on the use of other content such as images. This is due to the fact that the number of tagged image datasets related to disasters is very limited. Therefore, in this thesis, we aim to address this limitation by presenting a multimodal Turkish text and images dataset. Multimodal classification studies were carried out with the late fusion technique. Also, to achieve multimodal classification; a pre-trained LSTM model is used for classifying the text while a pre-trained CNN model is used for the visual content. Overall, concatenating both inputs in a multimodal learning architecture achieved an accuracy of 91.87%.