Visual-Graph-Memory

This is an official GitHub Repository for paper "Visual Graph Memory with Unsupervised Representation for Visual Navigation", which is accepted as a regular paper (poster) in ICCV 2021.

Setup

Requirements

The source code is developed and tested in the following setting.

Python 3.6
pytorch 1.4~1.7
habitat-sim 0.1.7 (commit version: ee75ba5312fff02aa60c04f0ad0b357452fc2edc)
habitat 0.1.7 (commit version: 34a4042c03d6596f1d614faa4891868ddaf81c04)

Please refer to habitat-sim and habitat-lab for installation.

To set the environment, run:

pip install -r requirements.txt

Habitat Data (Gibson, MP3D) Setup

Most of the scripts in this code build the environments assuming that the gibson/mp3d dataset is in habitat-lab/data/ folder.

The recommended folder structure of habitat-api (or habitat-lab):

habitat-api (or habitat-lab)
  └── data
      └── datasets
      │   └── pointnav
      │       └── gibson
      │           └── v1
      │               └── train
      │               └── val
      └── scene_datasets
          └── gibson_habitat
              └── *.glb, *.navmeshs

otherwise, you should edit the data path in these lines.

VGM Demonstration

To visualize the VGM generation, run:

python vgm_demo.py --gpu 0 --num-proc 2

This command will show the online VGM generation during random exploration. The rendering window will show the generated VGM and the observations as follows:

Note that the top-down map and pose information are only used for visualization, not for the graph generation.

Imitation Learning

Data generation
```
python collect_IL_data.py --ep-per-env 200 --num-procs 4 --split train --data-dir /path/to/save/data
```
This will generate the data for imitation learning. You can find some examples of the collected data in IL_data folder, and look into them with show_IL_data.ipynb.

Training

python train_bc.py --config configs/vgm.yaml --stop --gpu 0

Evaluation

Reinforcement Learning

The reinforcement learning code is highly based on habitat-lab/habitat_baselines. To train the agent with reinforcement learning (PPO), run:

python train_rl.py --config configs/vgm.yaml --version EXPERIMENT_NAME --diff hard --render --stop --gpu 0

Evaluation

We provide evaluation code and the pretrained model.

python evaluate_random.py --config configs/vgm.yaml --version-name test --eval-ckpt VGM_ILRL.pth --stop --diff hard

You can use "evaluate_dataset.py" to evaluate VGM on public image-goal nav dataset (https://github.com/facebookresearch/image-goal-nav-dataset)

git clone https://github.com/facebookresearch/image-goal-nav-dataset.git
python evaluate_dataset.py --config configs/vgm.yaml --version-name test --eval-ckpt VGM_ILRL.pth --stop --diff hard

In the above dataset, the provided pretrained model shows following performances.

Easy(SR)	Easy(SPL)	Medium(SR)	Medium(SPL)	Hard(SR)	Hard(SPL)	Overall(SR)	Overall(SPL)
0.76	0.40	0.76	0.56	0.62	0.49	0.71	0.48

zhoukang123/Visual-Graph-Memory