This repository provides code and data for our unpublished error analysis of an attention-based image description model. If you use this work, please cite:
@article{DBLP:journals/corr/MiltenburgE17,
author = {Emiel van Miltenburg and
Desmond Elliott},
title = {Room for improvement in automatic image description: an error analysis},
journal = {CoRR},
volume = {abs/1704.04198},
year = {2017},
url = {http://arxiv.org/abs/1704.04198},
archivePrefix = {arXiv},
eprint = {1704.04198},
timestamp = {Mon, 13 Aug 2018 16:48:09 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/MiltenburgE17},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
}
- Python 3.6 or higher.
- Flask 1.0.2
- Tabulate 0.8.2 (only for visualizing the results in our paper)
- Run
python congruency.py
- Go to http://127.0.0.1:5000/ in your browser to start annotating.
You will see a screen with several menu items:
- Home: This is the start page.
- Congruency: This page shows the coarse-grained annotation task.
- Load congruency data: This page allows you to load annotations from another annotator. To load the annotations from our main annotator, enter "annotations_emiel/congruency_data_final.json".
- Inspect congruent: This page allows you to leaf through the congruent descriptions.
- Inspect incongruent: This page allows you to leaf through the in congruent descriptions.
- Categorize incongruent: This page shows the second annotation task, which is to categorize to errors from the incongruent descriptions.
- As the file suggests, modify line 38 to annotate corrections, and save the file.
# MODIFY THIS LINE TO ANNOTATE A DIFFERENT CATEGORY:
categories_to_annotate = {'scene/event/location'}
- Run
python corrections.py
- Go to http://127.0.0.1:5000/ in your browser to start correcting the errors.
- Once you're done, the results will be stored in
./fixed/CATEGORY_fixed.txt
.
The file eval.sh
contains all commands to evaluate the results using the MS COCO evaluation scripts (also included in this repository).
It assumes that there is a Python 2 environment, created using Anaconda, called 'python2'. This is because the MS COCO evaluation scripts do not work well with Python 3.
To use this script, run: bash eval.sh
Run python generate_metrics_table.py
to print the table from our paper.