VISA

VISA is a dataset that consists of 40k Japanese-English parallel sentence pairs and corresponding video clips with the following key features:

The parallel sentences are subtitles from movies and TV episodes
The source subtitles are ambiguous, which means they have multiple possible translations with different meanings
We divide the dataset into Polysemy and Omission according to the cause of ambiguity

Examples:

Polysemy:

放せ！ --> Let me go!

Omission:

銃を持ってる。 --> I have a gun.

Splits:

Split	Train	Validation	test
Polysemy	18,666	1,000	1,000
Omission	17,214	1,000	1,000
Combined	35,880	2,000	2,000

Usage:

You can read json files to find the mapping from videos to parallel subtitle pairs.

Json Files Structure:

video_file_name: {  
    { "ja": Japanese_subtitle },  
    { "en": English_subtitle }  
}

Note:

Please, note that by downloading the dataset, you agree to the following conditions:

Do not re-distribute the dataset without our permission.
The dataset can only be used for research purposes. Any other use is explicitly prohibited.

Downloadable Features:

If you are interested in the video features of VISA, you can download them from the following links:

The I3D Features of VISA: http://lotus.kuee.kyoto-u.ac.jp/~yihang/dataset/VISA_i3d.zip
The RCNN Features of VISA: http://lotus.kuee.kyoto-u.ac.jp/~yihang/dataset/VISA_rcnn.zip

Citation:

If you find this dataset helpful, please cite our publication "VISA: An Ambiguous Subtitles Dataset for Visual Scene-Aware Machine Translation":

@inproceedings{li-etal-2022-visa,
    title = "{VISA}: An Ambiguous Subtitles Dataset for Visual Scene-aware Machine Translation",
    author = "Li, Yihang  and
      Shimizu, Shuichiro  and
      Gu, Weiqi  and
      Chu, Chenhui  and
      Kurohashi, Sadao",
    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference",
    month = jun,
    year = "2022",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2022.lrec-1.725",
    pages = "6735--6743",
}

Contact:

If you have any questions about this dataset, please contact liyh@nlp.ist.i.kyoto-u.ac.jp.

License:

GNU General Public License v3.0

ku-nlp/VISA