/Stereo-LiDAR-CCVNorm

Implementation of "Stereo and LiDAR Fusion using Stereo Matching Network with Conditional Cost Volume Normalization"

Primary LanguagePython

Stereo-LiDAR-CCVNorm

project page | paper | video

This is an implementation for our paper "3D LiDAR and Stereo Fusion using Stereo Matching Network with Conditional Cost Volume Normalization", IROS 2019.

Overview

Prerequisites

  • PyTorch
  • tqdm
  • tensorboardX
  • easydict
  • scikit-image
  • PIL
  • termcolor

How to Use

# KITTI Depth Completion
./data/
     |--kitti2017/
                 |--rgb/
                       |--2011_09_26/
                       |--2011_09_28/
                       |--2011_09_29/
                       |--2011_09_30/
                       |--2011_10_03/
                 |--depth/
                         |--train/
                         |--val
# KITTI Stereo 2015
./data/
      |--kitti_stereo/
                     |--data_scene_flow/
                                       |--training/
                                       |--testing/

            Afterward, setup a directory for all experiments. All you have to do in advance may look like this,

>> mkdir -p ${SOMEWHERE}/data/kitti2017 ${SOMEWHERE}data/kitti_stereo

# DOWNLOAD KITTI DEPTH COMPLETION
>> cd ${SOMEWHERE}/data/kitti2017
# reqest download from http://www.cvlibs.net/download.php?file=data_depth_annotated.zip
>> unzip data_depth_annotated.zip
# request the raw data downloader from http://www.cvlibs.net/download.php?file=raw_data_downloader.zip 
>> unzip raw_data_downloader
>> ./raw_data_downloader/raw_data_downloader.sh # NOTE: the raw data is extremely large (about 200GB), make sure you have enough disk space and prepared to wait for a long time

# DOWNLOAD KITTI STEREO 2015
>> cd ${SOMEWHERE}/data/kitti_stereo
# request download from http://www.cvlibs.net/download.php?file=data_scene_flow.zip
>> unzip data_scene_flow.zip

# SETUP REPO
>> git clone https://github.com/zswang666/Stereo-LiDAR-CCVNorm.git
>> cd Stereo-LiDAR-CCVNorm
>> mkdir -p data/kitti2017 data/kitti_stereo
>> ln -s ${SOMEWHERE}/data/kitti2017/data_depth_annotated data/kitti2017/depth
>> ln -s ${SOMEWHERE}/data/kitti2017 data/kitti2017/rgb
>> ln -s ${SOMEWHERE}/data/kitti_stereo data/kitti_stereo
>> mkdir exp
  • Training procedure (detailed configuration is specified in here):
>> vim misc/options.py # edit training configuration; see comments for more detailed explanation
>> python train.py
  • Testing prodedure:
# For KITTI Depth Completion
>> python test.py --model_cfg exp/test/test_options.py --model_path exp/test/ckpt/\[ep-00\]giter-0.ckpt --dataset kitti2017 --rgb_dir ./data/kitti2017/rgb --depth_dir ./data/kitti2015/depth
# For KITTI Stereo
>> python test.py --model_cfg exp/test/test_options.py --model_path exp/test/ckpt/\[ep-00\]giter-0.ckpt --dataset kitti2015 --root_dir ./data/kitti_stereo/data_scene_flow

Notes

  • Checkout model/ccvnorm.py for detailed implementation of our Conditional Cost Volume Normalization.
  • The training process will cost a lot of GPU memory. Please make sure you have a GPU with 16G or larger memory.
  • For testing (without gradient stored), 1080Ti (12G) is enough for a 256 x 960 image.

Results

Citation

@article{wang20193d,
  title={3D LiDAR and Stereo Fusion using Stereo Matching Network with Conditional Cost Volume Normalization},
  author={Wang, Tsun-Hsuan and Hu, Hou-Ning and Lin, Chieh Hubert and Tsai, Yi-Hsuan and Chiu, Wei-Chen and Sun, Min},
  journal={arXiv preprint arXiv:1904.02917},
  year={2019}
}