Serdar Yıldız* Abbas Memiş Songül Varlı
This paper presents a research study on the impact of deep machine translation on automatic Turkish image captioning. In the current literature of image processing, related studies on Turkish image captioning are quite limited. One of the main reasons why Turkish image captioning studies are quite limited is that large-scale datasets in Turkish for image captioning have not been constructed yet. In this study, an image caption set for Turkish was generated by using a recent deep machine translation model that becomes prominent with its high performance. In this context, for the MS COCO database, which is a commonly known and widely used image set, the original image captions written in English for images in this database were translated into Turkish using the NLLB (No Language Left Behind) deep machine translation model. In addition, a Turkish image captioning model based on the LSTM (Long Short-term Memory) which uses ResNet, ResNext and Swin deep learning structures as a backbone has also been evaluated on this derived Turkish image caption set. In performance evaluation tests, generally close performances were observed for all image encoder backbone models, and an average of 0.31 BLEU-1, 0.10 BLEU-2, 0.04 BLEU-3, 0.02 BLEU-4, 0.11 METEOR, 0.26 ROUGE-L and 0.04 CIDer values were measured. The related caption set created by using the NLLB deep machine translation model within the scope of the study has also been made available for general use over the web so that researchers working on similar topics can also benefit from it.
For the COCO dataset : COCO-Turkish
For the Flickr30k dataset, please visit the TRCaptionNet repository.
If you find our work helpful, please cite the following paper:
@INPROCEEDINGS{10286693,
author={Yıldız, Serdar and Memiş, Abbas and Çarlı, Songül},
booktitle={2023 8th International Conference on Computer Science and Engineering (UBMK)},
title={Automatic Turkish Image Captioning: The Impact of Deep Machine Translation},
year={2023},
volume={},
number={},
pages={414-419},
doi={10.1109/UBMK59864.2023.10286693}}