/Oscar

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Primary LanguagePythonMIT LicenseMIT

Oscar: Object-Semantics Aligned Pre-training for Vision-and-Language Tasks

Updates

05/28/2020: Released finetuned models on downstream tasks, please check MODEL_ZOO.md. 05/15/2020: Released pretrained models, datasets, and code for downstream tasks finetuning.

Introduction

This repository contains source code necessary to reproduce the results presented in the paper Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. We propose a new cross-modal pre-training method Oscar (Object-Semantics Aligned Pre-training). It leverages object tags detected in images as anchor points to significantly ease the learning of image-text alignments. We pre-train Oscar on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks. For more on this project, see the Microsoft Research Blog post.

Performance

Task t2i t2i i2t i2t IC IC IC IC NoCaps NoCaps VQA NLVR2
Metric R@1 R@5 R@1 R@5 B@4 M C S C S test-std test-P
SoTA_S 39.2 68.0 56.6 84.5 38.9 29.2 129.8 22.4 61.5 9.2 70.90 53.50
SoTA_B 48.4 76.7 63.3 87.0 39.5 29.3 129.3 23.2 73.1 11.2 72.54 78.87
SoTA_L 51.7 78.4 66.6 89.4 - - - - - - 73.40 79.50
----- --- --- --- --- --- --- --- --- --- --- --- ---
Oscar_B 54.0 80.8 70.0 91.1 40.5 29.7 137.6 22.8 78.8 11.7 73.44 78.36
Oscar_L 57.5 82.8 73.5 92.2 41.7 30.6 140.0 24.5 80.9 11.3 73.82 80.05
gain 5.8 4.4 6.9 2.8 2.2 1.3 10.7 1.3 7.8 0.5 0.42 0.55

t2i: text-to-image retrieval; i2t: image-to-text retrieval; IC: image captioning on COCO.

Download

We released pre-trained models and datasets for downstream tasks. Please check DOWNLOAD.md for details.

Installation

Check INSTALL.md for installation instructions.

Model Zoo

Check MODEL_ZOO.md for scripts to run each downstream tasks and the expected performance.

Citations

Please consider citing this paper if you use the code:

@article{li2020oscar,
  title={Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks},
  author={Li, Xiujun and Yin, Xi and Li, Chunyuan and Hu, Xiaowei and Zhang, Pengchuan and Zhang, Lei and Wang, Lijuan and Hu, Houdong and Dong, Li and Wei, Furu and Choi, Yejin and Gao, Jianfeng},
  journal={arXiv preprint arXiv:2004.06165},
  year={2020}
}

License

Oscar is released under the MIT license. See LICENSE for details.