EXPERT - Learning to Learn Words from Visual Scenes
Code from the paper Learning to Learn Words from Visual Scenes
Website of the project in expert.cs.columbia.edu
If you use the code, please cite the paper as:
@article{suris2019learning,
title={Learning to learn words from visual scenes},
author={Surís, Dídac and Epstein, Dave and Ji, Heng and Chang, Shih-Fu and Vondrick, Carl},
journal={arXiv preprint arXiv:1911.11237},
year={2019}
}
An example of command line execution can be found in scripts/run.sh
. To reproduce the numbers from the paper, please use the released pretrained models, and the scripts/test_*.sh
scripts.
Run python main.py --help
for information on arguments.
Be sure to have the external libraries in requirements.txt installed.
Data
We work with the Epic Kitchens dataset for this project. To run our code, you will need to download their images and annotations.
Specifically, the annotations directory has to contain:
- EPIC_train_object_labels.csv (provided by the Epic Kitchens dataset)
- EPIC_video_info.csv (provided by the Epic Kitchens dataset)
- splits.pth. File containing our train/test splits.
- processed_EPIC_train_action_labels
The path to this directory has to be introduced in the --annotation_root
argument. A compressed .tar.gz file with
these four files can be downloaded here.
The images directory has to be specified using --img_root
. It contains all the images with the following
subfolder structure: path_to_img_root/participant_id/vid_id/frame_{frame_id:010d}.jpg
. This is the default structure
if you download the data from the Epic Kitchens website
(download from here). For this project we only use
the RGB images, not flow information.
Pretrained models
The pretrained models reported in our paper can be found in the following links:
- Bert baseline
- Model with isolated attention
- Model with target-to-reference attention
- Model with via-vision attention
- Model with via-vision attention and input pointing
- Model with full attention
Each one of these is a .tar.gz file containing the files necessary to load the model (checkpoint_best.pth, config.json and tokenizer.pth).
To resume training or to test from one of these pretrained models, set the --resume
to True.
Extract the models under the /path/to/your/checkpoints
directory you introduce in
the --checkpoint_dir
argument. Refer to the specific model using the --resume_name
argument.