/lxmertcatt

Primary LanguagePythonMIT LicenseMIT

The Most Important Thing.

Our code is developed based on:

LXMERT: Learning Cross-Modality Encoder Representations from Transformers (https://github.com/airsplay/lxmert)

If you think our work is useful, please also cite their work!

Introduction

PyTorch code for the CVPR 2021 paper "Causal Attention for Vision-Language Tasks". PyTorch code for the CVPR 2021 paper "Causal Attention for Vision-Language Tasks". Slides of our EMNLP 2019 talk are avialable here. For experiment settings, like the pytorch version and GPU setting, please refer to LXMERT (https://github.com/airsplay/lxmert)

Results 36 RoI version

Split VQA GQA NLVR2
Local Validation 70.40% 60.90% 76.40%
Test-Dev 72.81% 60.84% 76.40% (Test-P)
Test-Standard 73.04% 61.17% 76.00% (Test-U)

Results 64 RoI version

Extracting more RoI visual features from an image will largely improve the performances!

Split VQA GQA NLVR2
Test-Dev 73.54% 61.87% 77.27% (Test-P)
Test-Standard 73.63% 62.07% 77.23% (Test-U)

Pre-training

Notice that this part is the same as LXMERT: https://github.com/airsplay/lxmert. We put them here for self-containing.

  1. Download the aggregated LXMERT dataset from MS COCO, Visual Genome, VQA, and GQA (around 700MB in total). The joint answer labels are saved in data/lxmert/all_ans.json.

    mkdir -p data/lxmert
    wget --no-check-certificate https://nlp1.cs.unc.edu/data/lxmert_data/lxmert/mscoco_train.json -P data/lxmert/
    wget --no-check-certificate https://nlp1.cs.unc.edu/data/lxmert_data/lxmert/mscoco_nominival.json -P data/lxmert/
    wget --no-check-certificate https://nlp1.cs.unc.edu/data/lxmert_data/lxmert/vgnococo.json -P data/lxmert/
    wget --no-check-certificate https://nlp1.cs.unc.edu/data/lxmert_data/lxmert/mscoco_minival.json -P data/lxmert/
  2. Download the detection features from MS COCO images from LXMERT.

    mkdir -p data/mscoco_imgfeat
    wget --no-check-certificate https://nlp1.cs.unc.edu/data/lxmert_data/mscoco_imgfeat/train2014_obj36.zip -P data/mscoco_imgfeat
    unzip data/mscoco_imgfeat/train2014_obj36.zip -d data/mscoco_imgfeat && rm data/mscoco_imgfeat/train2014_obj36.zip
    wget --no-check-certificate https://nlp1.cs.unc.edu/data/lxmert_data/mscoco_imgfeat/val2014_obj36.zip -P data/mscoco_imgfeat
    unzip data/mscoco_imgfeat/val2014_obj36.zip -d data && rm data/mscoco_imgfeat/val2014_obj36.zip
  3. Download the detection features for Visual Genome images.

    mkdir -p data/vg_gqa_imgfeat
    wget --no-check-certificate https://nlp1.cs.unc.edu/data/lxmert_data/vg_gqa_imgfeat/vg_gqa_obj36.zip -P data/vg_gqa_imgfeat
    unzip data/vg_gqa_imgfeat/vg_gqa_obj36.zip -d data && rm data/vg_gqa_imgfeat/vg_gqa_obj36.zip
  4. Test on a small split of the MS COCO + Visual Genome datasets:

    bash run/lxmert_pretrain.bash 0,1,2,3 --multiGPU --tiny
  5. Run on the whole MS COCO and Visual Genome related datasets (i.e., VQA, GQA, COCO caption, VG Caption, VG QA).

This part is ours:

The pre-training code is:

bash run/fsb2.bash 0,1,2,3 --multiGPU

After pre-training, the finetuning codes for VQA, GQA, and NLVE2 are:

bash run/vqa_finetuneft.bash 0 0.00004 0.00004
bash run/gqa_finetuneft.bash 0 0.000001 0.000001
bash run/nlvr2_ft.bash 0 0.00003 0.00003