Syllable-based image captioning model based on deep neural networks for image archives

Image captioning also known as automatic image description, refers to the process of automatically describing the content of an image by using natural language. In the field of image captioning, there are many models designed for English where computer vision and natural language processing techniques are combined. Direct adaptation of these models to Turkish is not possible due to the agglutinative structure of the Turkish language. In our study, we propose a syllable-based image captioning model to better understand the structure of the language. The proposed model follows an encoder-decoder architecture, utilizing both CNN and LSTM. Three seperate datasets were used to compare the performance of the proposed syllable-based model with word-level and baseword/subword-level models. The first and second datasets are the Flickr8k and Flickr30k dataset, which is publicly accessible for researchers, while the other one is a dataset that we created. The performance of the proposed approach was evaluated by using the BLEU metric, and we found that the syllable-based model outperformed the other two models in all datasets. To the best of our knowledge, there is no existing work on syllable-based image captioning, making our study significant as the first work.