This repo holds the implementation code and pre-trained weights for the following paper:
Runjia Li*, Shuyang Sun*, Mohamed Elhoseiny and Philip Torr, OxfordTVG-HIC: Can Machine Make Humorous Captions from Images? Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
This paper presents Oxford HIC, a large-scale dataset for humour generation and understanding. Humour is an abstract, subjective, and context-dependent cognitive construct involving several cognitive factors, making it a challenging task to generate and interpret. Hence, humour generation and understanding can serve as a new task for evaluating the ability of deep-learning methods to process abstract and subjective information. Due to the scarcity of data, humour-related generation tasks such as captioning remain under-explored. To address this gap, Oxford HIC offers approximately 2.9M image-text pairs with humour scores to train a generalizable humour captioning model. Contrary to existing captioning datasets, Oxford HIC features a wide range of emotional and semantic diversity resulting in out-of-context examples that are particularly conducive to generating humour. Moreover, Oxford HIC is curated devoid of offensive content. We also show how Oxford HIC can be leveraged for evaluating the humour of a generated text. Through explainability analysis of the trained models, we identify the visual and linguistic cues influential for evoking humour prediction (and generation). We observe qualitatively that these cues are aligned with the benign violation theory of humour in cognitive psychology.
Download the dataset from Oxford_HIC or directly running the following scripts to download meta-data and images simultaneously (make sure you have Gdown in your environment).
cd data
bash download_data.sh
This will automatically downlod images and captions to data/hic_data
There are some problems with the weight. We will fix the issue soon.
If you found this repository useful, please consider citing:
@InProceedings{Li_2023_ICCV,
author = {Li, Runjia and Sun, Shuyang and Elhoseiny, Mohamed and Torr, Philip},
title = {OxfordTVG-HIC: Can Machine Make Humorous Captions from Images?},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2023},
pages = {20293-20303}
}