LiCROcc

This repository contains the implementation of the paper.

If you find our work useful, Please give us a star 🌟!

LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera
Yukai Ma^1,2, Jianbiao Mei^1,2, Xuemeng Yang², Licheng Wen², Weihua Xu¹, Jiangning Zhang¹, Xingxing Zuo³, Botian Shi^{2,^}, Yong Liu^{1,^}
¹ZJU ²PJLab³TUM
^{^}Corresponding Authors

News

🎉 [2024/11/06] LiCROcc was accepted by RAL!!!

Getting Started

Installation

# Create a conda virtual environment and activate it.
conda create -n licrocc python=3.8 -y
conda activate licrocc
# Install torch
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f ttps://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt
# Install mmdet3d
git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d
git checkout v0.17.1 # Other versions may not be compatible.
pip install -v -e .
# 
cd ../projects
pip install -v -e .

Prepare Dataset

-Please refer to OpenOccupancy to prepare nuScenes dataset.

Please refer to CRN to generate radar point cloud in BEV view.

python scripts/gen_radar_bev.py  # accumulate sweeps and transform to LiDAR coords

Folder structure:

LiCROcc
├── data/
│   ├── nuscenes/
│   │   ├── maps/
│   │   ├── samples/
│   │   ├── sweeps/
│   │   ├── lidarseg/
│   │   ├── v1.0-test/
│   │   ├── v1.0-trainval/
│   │   ├── nuscenes_occ_infos_train.pkl
│   │   ├── nuscenes_occ_infos_val.pkl
│   │   ├── radar_bev_filter/
│   ├── nuScenes-Occupancy/
│   ├── depth_gt/

Run and Eval

Download Teacher model here

Train RC-LiCROcc

./tools/dist_train.sh ./projects/configs/ssc_rs/ssc_rs_base_nuscenes_LC2LR123.py N_GPUs

Train R-LiCROcc

./tools/dist_train.sh ./projects/configs/ssc_rs/ssc_rs_base_nuscenes_LC2radar12.py N_GPUs

If an error is reported during training：

TypeError: FormatCode() got an unexpected keyword argument 'verify'

Simply remove ‘verify=True’

Eval RC-LiCROcc

./tools/dist_test.sh ./projects/configs/ssc_rs/ssc_rs_base_nuscenes_LC2LR123.py ./path/to/ckpts.pth N_GPUs

Eval R-LiCROcc

./tools/dist_test.sh ./projects/configs/ssc_rs/ssc_rs_base_nuscenes_LC2radar12.py ./path/to/ckpts.pth N_GPUs

Visualization

Temporarily only support saving occupancy predictions (refer to MonoScene for visualization tools)

./tools/dist_test.sh $PATH_TO_CFG $PATH_TO_CKPT $GPU_NUM --show --show-dir $PATH

Model Zoo

Subset	Checkpoint	Logs	Note
RC-LiCROcc	link	link	train on 8 A100
R-LiCROcc	link	link	train on 8 A100

Cite Us

@misc{ma2024licroccteachradaraccurate,
title={LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera}, 
author={Yukai Ma and Jianbiao Mei and Xuemeng Yang and Licheng Wen and Weihua Xu and Jiangning Zhang and Botian Shi and Yong Liu and Xingxing Zuo},
year={2024},
eprint={2407.16197},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2407.16197}, 
}

Credit

We adopt the following open-sourced projects:

HR-zju/LiCROcc