/review_net

Review Network for Caption Generation

Primary LanguageLuaMIT LicenseMIT

Review Network for Caption Generation

Image Captioning on MSCOCO

You can use the code in this repo to genearte a MSCOCO evaluation server submission with CIDEr=0.96+ with just a few hours.

No fine-tuning required. No fancy tricks. Just train three end-to-end review networks and do an ensemble.

  • Feature extraction: 2 hours in parallel
  • Single model training: 6 hours
  • Ensemble model training: 30 mins
  • Beam search for caption generation: 3 hours in parallel

Below is a comparison with other state-of-the-art systems (with according published papers) on the MSCOCO evaluation server:

Model BLEU-4 METEOR ROUGE-L CIDEr Fine-tuned Task specific features
Attention 0.537 0.322 0.654 0.893 No No
MS Research 0.567 0.331 0.662 0.925 No Yes
Google NIC 0.587 0.346 0.682 0.946 Yes No
Semantic Attention 0.599 0.335 0.682 0.958 No Yes
Review Net 0.597 0.347 0.686 0.969 No No

In the diretcory image_caption_online, you can use the code therein to reproduce our evaluation server results.

In the directory image_caption_offline, you can rerun experiments in our paper using offline evaluation.

Code Captioning

Predicting comments for a piece of source code is another interesting task. In the repo we also release a dataset with train/dev/test splits, along with the code of a review network.

Check out the directory code_caption.

Below is a comparison with baselines on the code captioning dataset:

Model LLH CS-1 CS-2 CS-3 CS-4 CS-5
LSTM Language Model -5.34 0.2340 0.2763 0.3000 0.3153 0.3290
Encoder-Decoder -5.25 0.2535 0.2976 0.3201 0.3367 0.3507
Encoder-Decoder (Bidir) -5.19 0.2632 0.3068 0.3290 0.3442 0.3570
Attentive Encoder-Decoder (Bidir) -5.14 0.2716 0.3152 0.3364 0.3523 0.3651
Review Net -5.06 0.2889 0.3361 0.3579 0.3731 0.3840

References

This repo contains the code and data used in the following paper:

Review Networks for Caption Generation

Zhilin Yang, Ye Yuan, Yuexin Wu, Ruslan Salakhutdinov, William W. Cohen

NIPS 2016