Depth Estimation from Camera Image and mmWave Radar Point Cloud

Pytorch implementation of Depth Estimation from Camera Image and mmWave Radar Point Cloud

CVPR 2023

Models have been tested on Ubuntu 20.04 using Python 3.8, Pytorch 1.10.2+cu113

If you use this work, please cite our paper:

@inproceedings{singh2023depth,
  title={Depth Estimation From Camera Image and mmWave Radar Point Cloud},
  author={Singh, Akash Deep and Ba, Yunhao and Sarker, Ankur and Zhang, Howard and Kadambi, Achuta and Soatto, Stefano and Srivastava, Mani and Wong, Alex},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={9275--9285},
  year={2023}
}

Setting up your virtual environment

We will create a virtual environment with the necessary dependencies

virtualenv -p /usr/bin/python3 cam_rad_fusion
source cam_rad_fusion/bin/activate

pip install -r requirements.txt

Setting up dataset for RadarNet

Note: Run all bash scripts from the root directory.

We use the nuScenes dataset that can be downloaded here.

Please create a folder called data and place the downloaded nuScenes dataset into it.

Generate the panoptic segmentation masks using the following:

python setup/gen_panoptic_seg.py

Then run the following bash script to generate the preprocessed dataset for training and testing:

bash setup_dataset_nuscenes.sh
bash setup_dataset_nuscenes_test.sh

This will generate the training dataset in a folder called data/nuscenes_derived

Training RadarNet

To train RadarNet on the nuScenes dataset, you may run

bash train_radarnet_nuscenes.sh

To monitor your training progress, you may use Tensorboard

tensorboard --logdir trained_radarnet/<model_name>

Setting up dataset for FusionNet

Once RadarNet training is complete, you can run:

bash setup_dataset_nuscenes_radarnet.sh
bash setup_dataset_nuscenes_radarnet_test.sh

Training FusionNet

To train RadarNet on the nuScenes dataset, you may run

bash train_fusionnet_nuscenes.sh

To monitor your training progress, you may use Tensorboard

tensorboard --logdir trained_fusionnet/<model_name>

Downloading our pre-trained models

Our pretrained models are available here for download:

https://drive.google.com/drive/folders/1ZG-2UG67VcYhWTyHdUzTHIzdfCpsG_1Q?usp=sharing

Evaluating our models

To evaluate the pretrained Fusionnet on the nuScenes dataset, you may run:

bash run_fusionnet_nuscenes_test.sh

You may replace the restore_path and output_path arguments to evaluate your own checkpoints

License and disclaimer

This software is property of the UC Regents, and is provided free of charge for research purposes only. It comes with no warranties, expressed or implied, according to these terms and conditions. For commercial use, please contact UCLA TDG.

nesl/radar-camera-fusion-depth