Visual-Graph-Memory

This is an official GitHub Repository for paper "Visual Graph Memory with Unsupervised Representation for Visual Navigation", which is accepted as a regular paper (poster) in ICCV 2021.

Setup

Requirements

The source code is developed and tested in the following setting.

Python 3.6
pytorch 1.4~1.7
habitat-sim 0.1.7 (commit version: ee75ba5312fff02aa60c04f0ad0b357452fc2edc)
habitat 0.1.7 (commit version: 34a4042c03d6596f1d614faa4891868ddaf81c04)

Please refer to habitat-sim and habitat-lab for installation.

To set the environment, run:

pip install -r requirements.txt

Habitat Data (Gibson, MP3D) Setup

Most of the scripts in this code build the environments assuming that the gibson/mp3d dataset is in habitat-lab/data/ folder.

The recommended folder structure of habitat-api (or habitat-lab):

habitat-api (or habitat-lab)
  └── data
      └── datasets
      │   └── pointnav
      │       └── gibson
      │           └── v1
      │               └── train
      │               └── val
      └── scene_datasets
          └── gibson_habitat
              └── *.glb, *.navmeshs

otherwise, you should edit the data path in these lines.

VGM Demonstration

To visualize the VGM generation, run:

python vgm_demo.py --gpu 0 --num-proc 2

This command will show the online VGM generation during random exploration. The rendering window will show the generated VGM and the observations as follows:

Note that the top-down map and pose information are only used for visualization, not for the graph generation.

Imitation Learning

Data generation
```
python collect_IL_data.py --ep-per-env 200 --num-procs 4 --split train --data-dir /path/to/save/data
```
This will generate the data for imitation learning. You can find some examples of the collected data in IL_data folder, and look into them with show_IL_data.ipynb.

Training

python train_bc.py --config configs/vgm.yaml --stop --gpu 0

Evaluation

Reinforcement Learning

The reinforcement learning code is highly based on habitat-lab/habitat_baselines. To train the agent with reinforcement learning (PPO), run:

python train_rl.py --config configs/vgm.yaml --version EXPERIMENT_NAME --diff hard --render --stop --gpu 0