Auxiliary Tasks and Exploration Enable ObjectNav

Joel Ye, Dhruv Batra, Abhishek Das, and Erik Wijmans.

Project Site: https://joel99.github.io/objectnav/

This repo is a code supplement for the paper "Auxiliary Tasks and Exploration Enable ObjectNav." The code is a fork of Habitat-Lab and borrows heavily from habitat-pointnav-aux.

Overview

The primary code contributions from the paper are located in:

Auxiliary Tasks: auxiliary_tasks.py and supervised_auxiliary_tasks.py
Exploration Reward: coverage.py
Tether logic (denoted as "split" in code): rollout_storage.py, ppo.py, multipolicy.py
Experimental Configurations: habitat_baselines/config/objectnav/full/*.yaml
Misellaneous business logic: ppo_trainer.py
Scripts and Analysis: ./scripts/*

Requirements

The code is known to be compatible with Python 3.6, Habitat Lab v0.1.5 and Habitat Sim v0.1.6 (headless). We recommend you use conda to initialize the code as follows:

Install habitat-sim. In the habitat-sim repo, run:

conda create -n <ENV_NAME> python=3.6 cmake=3.14.0
conda activate <ENV_NAME>
pip install -r requirements.txt
conda install habitat-sim=0.1.6 headless -c conda-forge -c aihabitat

Then, in this repo, run:

conda env update -f environment.yml --name <ENV_NAME>

Note: If you run into a pip error along the lines of a missing metadata file, it is likely an improperly cleaned package (e.g. numpy-1.19.5), just remove it. Note 2: You may also need to run python setup.py develop --all as in the habitat-lab installation process.

Data

The experiments were done on the Matterport3D dataset. Please refer to habitat-sim for the scene dataset and habitat-lab for the ObjectNav dataset.

Pre-trained Weights

You can download pretrained agent and rednet weights here. We provide the best checkpoint for agents as measured by their performance with RedNet segmentation:

base-full.31.pth: 6-action agent.
base4-full.33.pth: 4-action agent.
split_clamp.31.pth: 6 + Tether agent.
rednet_semmap_mp3d_tuned.pth: RedNet weights.

Training

We train our models for around 8 GPU-weeks, on a SLURM-managed cluster. On such a cluster, you can launch your run with:

sbat ./scripts/ddppo_8gpu.sh <args>

With the appropriate configuration file. The code in run.py is set up to parse out the arguments, so to train with the configuration .../objectnav/full/base.yaml, write:

sbat ./scripts/ddppo_8gpu.sh base full

On a machine without SLURM, see, e.g. ./scripts/train_no_ckpt.sh.

Evaluation

You can evaluate a trained checkpoint by configuring the arguments in ./scripts/eval_on.sh. This forwards to habitat_baselines/run.py, which accepts a number of flags. For example:

Add a -v flag to generate videos of your agent.
Add a -ra flag to record trajectories and hidden states for subsequent analysis.

See run.py for all arguments. Note that an evaluation on the full validation split can take a while (>1 GPU-day).

Citation

If you use this work, you can cite it as

@misc{ye2021auxiliary,
      title={Auxiliary Tasks and Exploration Enable ObjectNav},
      author={Joel Ye and Dhruv Batra and Abhishek Das and Erik Wijmans},
      year={2021},
      eprint={2104.04112},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Additional Notes

The 300-episode subset mentioned throughout analysis was created by sampling episodes uniformly across the scenes in train and val splits, using scripts/dataset_slicer.py
The .py notebooks in scripts/ start with a comment denoting their use.
Though the configurations default to mixed precision training, it is not particularly faster in our experiments.
The UNLICENSE applies to this project's changes on top of the habitat-lab codebase.

joel99/objectnav