Multimodal representation learning for synchronized speech and videos

The amount of multimedia data has been increased rapidly in recent years. While this data growth enables multimodal neural network based studies, it has also resulted in a need for efficient storage and retrieval systems for multimodal data. In this thesis, different data fusion schemes are examined to see the benefits of the use of different data sources. Proposed fusion schemes differ in their stages in which the data fusion is performed. Additionally, several representation learning methods are investigated for efficient data storage and retrieval systems. Representations are generated in such a way that they reflect the distance between the represented data segments according to a certain distance metric. A joint representation and distance metric learning scheme is also considered for a performance gain. Several deep neural network models are designed for representation learning and data fusion, and their performances are evaluated with the same-different word-discrimination and phone classification tasks, respectively. Experiments are performed on two different multimodal data sets; USC-TIMIT rtMRI and Signed Turkish broadcast news. Outcomes of the experiments show that the data fusion indeed brings a performance improvement over unimodal approaches, and performing fusion in earlier stages yields better results than fusing the data in later stages. Additionally, the proposed methods for the representation learning outperform the corresponding baseline systems in the same-different word-discrimination task. Therefore, generated representations of video and audio segments can be considered as an important step towards a fast cross-modal query-by-sign search system.