EmbodiedQA

Code for the paper

Embodied Question Answering
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, Dhruv Batra
arxiv.org/abs/1711.11543
CVPR 2018 (Oral)

In Embodied Question Answering (EmbodiedQA), an agent is spawned at a random location in a 3D environment and asked a question (for e.g. "What color is the car?"). In order to answer, the agent must first intelligently navigate to explore the environment, gather necessary visual information through first-person vision, and then answer the question ("orange").

This repository provides

Pretrained CNN for House3D
Code for generating EQA questions
- EQA v1: location, color, place preposition
- EQA v1-extended: existence, logical, object counts, room counts, distance comparison
Code to train and evaluate navigation and question-answering models
- independently with supervised learning on shortest paths
- jointly using reinforcement learning

If you find this code useful, consider citing our work:

@inproceedings{embodiedqa,
  title={{E}mbodied {Q}uestion {A}nswering},
  author={Abhishek Das and Samyak Datta and Georgia Gkioxari and Stefan Lee and Devi Parikh and Dhruv Batra},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2018}
}

Setup

virtualenv -p python3 .env
source .env/bin/activate
pip install -r requirements.txt

Download the SUNCG dataset and install House3D.

NOTE: This code uses a fork of House3D with a few changes to support arbitrary map discretization resolutions.

Question generation

Questions for EmbodiedQA are generated programmatically, in a manner similar to CLEVR (Johnson et al., 2017).

NOTE: Pre-generated EQA v1 questions are available for download here.

Generating questions for all templates in EQA v1, v1-extended

cd data/question-gen
./run_me.sh MM_DD

List defined question templates

from engine import Engine

E = Engine()
for i in E.template_defs:
    print(i, E.template_defs[i])

Generate questions for a particular template (say `location`)

from house_parse import HouseParse
from engine import Engine

Hp = HouseParse(dataDir='/path/to/suncg')
Hp.parse('0aa5e04f06a805881285402096eac723')

E = Engine()
E.cacheHouse(Hp)
qns = E.executeFn(E.template_defs['location'])

print(qns[0]['question'], qns[0]['answer'])
# what room is the clock located in? bedroom

Pretrained CNN

We trained a shallow encoder-decoder CNN from scratch in the House3D environment, for RGB reconstruction, semantic segmentation and depth estimation. Once trained, we throw away the decoders, and use the encoder as a frozen feature extractor for navigation and question answering. The CNN is available for download here:

wget https://www.dropbox.com/s/ju1zw4iipxlj966/03_13_h3d_hybrid_cnn.pt

The training code expects the checkpoint to be present in training/models/.

Supervised Learning

Download and preprocess the dataset

Download EQA v1 and shortest path navigations:

wget https://www.dropbox.com/s/6zu1b1jzl0qt7t1/eqa_v1.json
https://www.dropbox.com/s/lhajthx7cdlnhns/a-star-500.zip
unzip a-star-500.zip

If this is the first time you are using SUNCG, you will have to clone and use the SUNCG toolbox to generate obj + mtl files for the houses in EQA.

NOTE: Shortest paths have been updated. Earlier we computed shortest paths using a discrete grid world, but we found that these shortest paths were sometimes innacurate. Old shortest paths are [here] (https://www.dropbox.com/s/vgp2ygh1bht1jyb/shortest-paths.zip)

cd utils
python make_houses.py \
    -eqa_path /path/to/eqa.json \
    -suncg_toolbox_path /path/to/SUNCGtoolbox \
    -suncg_data_path /path/to/suncg/data_root

Preprocess the dataset for training

cd training
python utils/preprocess_questions_pkl.py \
    -input_json /path/to/eqa_v1.json \
    -shortest_path_dir /path/to/shortest/paths/v3 \
    -output_train_h5 data/train.h5 \
    -output_val_h5 data/val.h5 \
    -output_test_h5 data/test.h5 \
    -output_data_json data/data.json \
    -output_vocab data/vocab.json

Visual question answering

Update pretrained CNN path in models.py.

python train_vqa.py -to_log 1 -input_type ques,image -identifier ques-image

This model computes question-conditioned attention over last 5 frames from oracle navigation (shortest paths), and predicts answer. Assuming shortest paths are optimal for answering the question -- which is predominantly true for most questions in EQA v1 (location, color, place preposition) with the exception of a few location questions that might need more visual context than walking right up till the object -- this can be thought of as an upper bound on expected accuracy, and performance will get worse when navigation trajectories are sampled from trained policies.

Navigation

Download potential maps for evaluating navigation and training with REINFORCE.

wget https://www.dropbox.com/s/53edqtr04jts4q0/target-obj-conn-maps-500.zip

Planner-controller policy

python train_nav.py -to_log 1 -model_type pacman -identifier pacman

REINFORCE

python train_eqa.py -to_log 1 \
    -nav_checkpoint_path /path/to/nav/ques-image-pacman/checkpoint.pt \
    -ans_checkpoint_path /path/to/vqa/ques-image/checkpoint.pt \
    -identifier ques-image-eqa

Changelog

09/07

We added the baseline models from the CVPR paper (Reactive and LSTM).
With the LSTM model, we achieved d_T values of: 0.74693/3.99891/8.10669 on the test set for d equal to 10/30/50 respectively training with behavior cloning (no reinforcement learning).
We also updated the shortest paths to fix an issue with the shortest path algorithm we initially used. Code to generate shortest paths is here.

06/13

This code release contains the following changes over the CVPR version

Larger dataset of questions + shortest paths
Color names as answers to color questions (earlier they were hex strings)

Acknowledgements

Parts of this code are adapted from pytorch-a3c by Ilya Kostrikov
Lisa Anne Hendricks and Licheng Yu helped with running / testing / debugging code prior to release

License

BSD

himanshisyadav/EmbodiedQA