zhaoc5/Grounding-REVERIE-Challenge

Official REVERIE Grounding Model of REVERIE Challenge @ CSIG 2022

Grounding-REVERIE-Challenge

This repository is the official REVERIE Referring Expression Grounding model of REVERIE-Challenge. This code derives from UNITER, fine-tuned on REVERIE data. If you have any question, please do not hesitate to contact me, chongyang.zhao@adelaide.edu.au.

Requirements

You need Docker to run the code. For more details, please refer to UNITER for further README information. We tested on Ubuntu 20.04 and Nvidia 2080ti.

• NVIDIA Container Toolkit
• Docker group membership

Data Preparation

Note that BBoxes_v2 files have two different forms of organization, so you need download Grounding_BBoxes_v2, which is different from the one used by HOP-REVERIE-Challenge.

Download the REVERIE data, Grounding_BBoxes_v2 and object features (reverie_obj_feats_v2.pkl or reverie_bbox_feat_v2_caffe.zip, code: nubg), both of them are extracted using BUTD Faster R-CNN trained on Visual Genome, and organise data like below:

|- Grounding-REVERIE-Challenge
    |- Downloads
        |- REVERIE
        |   |- REVERIE_val_seen.json
        |   |- REVERIE_val_unseen.json
        |   |- REVERIE_test.json
        |                     
        |- Grounding_BBoxes_v2
        |   |- 1LXtFkjw3qL_0b22fa63d0f54a529c525afbf2e8bb25.json
        |   |- 1LXtFkjw3qL_0b302846f0994ec9851862b1d317d7f2.json
        |   ...           
        |   |- zsNo4HB9uLZ_faad06c7cb2b4a6f9220e7f6f87c800b.json
        |
        |- reverie_bbox_feat_v2_caffe (option 1)
        |   |- 1LXtFkjw3qL_0b22fa63d0f54a529c525afbf2e8bb25_00.h5
        |   |- 1LXtFkjw3qL_0b22fa63d0f54a529c525afbf2e8bb25_01.h5
        |   ...           
        |   |- zsNo4HB9uLZ_faad06c7cb2b4a6f9220e7f6f87c800b_35.h5
        |
        |- reverie_obj_feats_v2.pkl (option 2)

# Image object features preparation
# Make sure the parameters in script are correct
# 
# Support .h5 and .pkl files
# .h5:
# feats_path: "Downloads/reverie_bbox_feat_v2_caffe"
# feats_format: "h5"
# .pkl:
# feats_path: "Downloads/reverie_obj_feats_v2.pkl"
# feats_format: "pkl"

bash img_data_preparation.sh

Pre-trained Model

• Grounding pre-trained model weights: weights/ckpt/model_epoch_best.pt
  • Download the model_epoch_best.pt from here.

Usage

1.  # Docker image should be automatically pulled
   
    cd Grounding-REVERIE-Challenge
    bash launch_container.sh

2.  # Object id prediction 
    # Make sure the parameters in the script are correct
    #
    # Navigation Results:
    # input_nav_dir: "input_nav_dir"
    #
    # Grounding Results (Navigation Results + predObjId):
    # output_dir: "submit_file"
    #
    # reverie_dir: "Downloads/REVERIE"
    # boxes_dir: "Downloads/Grounding_BBoxes_v2"
    #
    # We support multi-GPU inference
    # NUM_GPUS=4
   
    bash eval.sh