/image-paragraph-captioning

PyTorch implementation of Training for Diversity in Image Paragraph Captioning (EMNLP 2018)

Primary LanguagePython

Training for Diversity in Image Paragraph Captioning

This repository includes a PyTorch implementation of Training for Diversity in Image Paragraph Captioning. Our code is based on Ruotian Luo's implementation of Self-critical Sequence Training for Image Captioning, available here..

Requirements

  • Python 2.7 (because coco-caption does not support Python 3)
  • PyTorch 0.4 (with torchvision)
  • cider (already included as a submodule)
  • coco-caption (already included as a submodule)

If training from scratch, you also need:

  • spacy (to tokenize words)
  • h5py (to store features)
  • scikit-image (to process images)

To clone this repository with submodules, use:

  • git clone --recurse-submodules https://github.com/lukemelas/image-paragraph-captioning

Train your own network

Download and preprocess cations

  • Download captions:
    • Run download.sh in data/captions
  • Preprocess captions for training (part 1):
    • Download spacy English tokenizer with python -m spacy download en
    • First, convert the text into tokens: cd scripts && python prepro_text.py
    • Next, preprocess the tokens into a vocabulary (and map infrequent words to an UNK token) with the following command. Note that image/vocab information is stored in data/paratalk.json and caption data is stored in data/paratalk\_label.h5
python scripts/prepro_labels.py --input_json data/captions/para_karpathy_format.json --output_json data/paratalk.json --output_h5 data/paratalk
  • Preprocess captions into a coco-captions format for calculating CIDER/BLEU/etc:
    • Run scripts/prepro\_captions.py
    • There should be 14,575/2487/2489 images and annotations in the train/val/test splits
    • Uncomment line 44 ((Spice(), "SPICE")) in coco-caption/pycocoevalcap/eval.py to disable Spice testing
  • Preprocess ngrams for self-critical training:
python scripts/prepro_ngrams.py --input_json data/captions/para_karpathy_format.json --dict_json data/paratalk.json --output_pkl data/para_train --split train
  • Extract image features using an object detector
    • We make pre-processed features widely available:
      • Download and extract parabu_fc and parabu_att from here into data/bu_data
    • Or generate the features yourself:
      • Download the Visual Genome Dataset
      • Apply the bottom-up attention object detector here made by Peter Anderson.
      • Use scripts/make_bu_data.py to convert the image features to .npz files for faster data loading

Train the network

As explained in Self-Critical Sequence Training, training occurs in two steps:

  1. The model is trained with a cross-entropy loss (~30 epochs)
  2. The model is trained with a self-critical loss (30+ epochs)

Training hyperparameters may be accessed with python train.py --help.

A reasonable set of hyperparameters is provided in train_xe.sh (for cross-entropy) and train_sc.sh (for self-critical).

mkdir log_xe
./train_xe.sh

You can then copy the model:

./scripts/copy_model.sh xe sc

And train with self-critical:

mkdir log_sc
./train_xe.sh

Pretrained Network

You can download a pretrained captioning model here.

Citation

In case you would like to cite our paper/code (no obligation at all):

@article{melaskyriazi2018paragraph, 
  title={Training for diversity in image paragraph captioning},
  author={Melas-Kyriazi, Luke and Rush, Alexander and Han, George},
  journal={EMNLP},
  year={2018}
}

And Ruotian Luo's code, on which this repo is built:

@article{luo2018discriminability,
  title={Discriminability objective for training descriptive captions},
  author={Luo, Ruotian and Price, Brian and Cohen, Scott and Shakhnarovich, Gregory},
  journal={CVPR},
  year={2018}
}