Behind the Curtain: Learning Occluded Shapes for 3D Object Detection (AAAI-2022)

Reference

Please cite our paper if you are interested to use this implementation,
Behind the Curtain: Learning Occluded Shapes for 3D Object Detection.

@article{xu2021behind,
  title={Behind the Curtain: Learning Occluded Shapes for 3D Object Detection},
  author={Xu, Qiangeng and Zhong, Yiqi and Neumann, Ulrich},
  journal={arXiv preprint arXiv:2112.02205},
  year={2021}
}

The implementatin is also inspired by the ICCV-2021 paper,
SPG: Unsupervised domain adaptation for 3d object detection via semantic point generation.

@inproceedings{xu2021spg,
  title={Spg: Unsupervised domain adaptation for 3d object detection via semantic point generation},
  author={Xu, Qiangeng and Zhou, Yin and Wang, Weiyue and Qi, Charles R and Anguelov, Dragomir},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={15446--15456},
  year={2021}
}

Acknowledgement

Our model, BtcDet, is implemented based on [OpenPcdet 0.3.0]. We thank Shaohuai Shi for the discussion during our implementation.

Installation

Requirements

All the codes are tested in the following environment:

Linux (tested on Ubuntu 16.04, should be able to work on 18.04)
Python 3.6+
PyTorch 1.1 or higher (tested on PyTorch 1.7, 1.8.1, 1.9, 1.10)
CUDA 9.0 or higher (PyTorch 1.3+ needs CUDA 9.2+, tested on CUDA 10.2)
spconv v1.2.1 (commit fad3000249d27ca918f2655ff73c41f39b0f3127)

Install

b. Install the dependent libraries as follows:

Install the dependent python libraries:

pip install -r requirements.txt

if using conda, using:

conda install -f requirement.yml
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge

follow this link to install cudnn https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html

Install the SparseConv library, we use the implementation from [spconv].
- The very recent spconv 2.+ has significant interface changes and some issues in operators. Therefore, we do not support it for now.
- If you use PyTorch 1.1, then make sure you install the spconv v1.0 with (commit 8da6f96) instead of the latest one.
- If you use PyTorch 1.3+, then you need to install the spconv v1.2. As mentioned in v1.2.1 of spconv (https://github.com/traveller59/spconv/tree/v1.2.1)
```
git clone -b v1.2.1  https://github.com/traveller59/spconv.git --recursive

cd spconv

sudo apt-get install libboost-all-dev

python setup.py bdist_wheel

(if having problems about THC/THCNumerics, see this repo: https://github.com/traveller59/spconv/issues/464)

cd ./dist 
```
Inside the dist directory, use pip to install generated whl file.
```
pip install spconv-1.2.1-{your system info}.whl
```
After that, you should first get out of the spconv directory, then do python import spconv to see if you installed it correctly.

c. Install the operators in BtcDet by running the following command:

cd BtcDet
python setup.py develop

Preparation

Use Our Preprocessed Data:

you can use our generated kitti's data including the generated complete object points, download it [here (about 31GBs)] and put the zip file inside data/kitti/ and unzip it as detection3d directory.

Alternatively, Generate Data by Yourself:

####KITTI Dataset

Please download the official KITTI 3D object detection dataset and organize the downloaded files as follows (the road planes could be downloaded from [road plane], which are optional for data augmentation in the training):

BtcDet
├── data
│   ├── kitti
    │   │   │──detection3d  │── ImageSets
                    │   │   │── training
                    │   │   │   ├──calib & velodyne & label_2 & image_2 & (optional: planes) & (optional: depth_2)
                    │   │   │── testing
                    │   │   │   ├──calib & velodyne & image_2

Generate the data infos by running the following command:

python -m btcdet.datasets.kitti.kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/kitti_dataset.yaml

Generate Approximated complete object points:

(Under BtcDet directory, execute:)

python -m btcdet.datasets.multifindbestfit

Run Training:

cd tools/

Single gpu training

mkdir output

mkdir output/kitti_car

python train.py --cfg_file ./cfgs/model_configs/btcdet_kitti_car.yaml --output_dir ../output/kitti_car/ --batch_size 2 --gpu_str "0"

Multi gpu training, assuming you have 4 gpus:

bash scripts/dist_train.sh 4  --batch_size 8 --gpu_str "0,1,2,3" --cfg_file ./cfgs/model_configs/btcdet_kitti_car.yaml --output_dir ../output/kitti_car/

Run Testing:

cd tools/

Single gpu testing for all saved checkpoints, assuming you have 4 gpus:

python test.sh 4 --eval-all --cfg_file ./cfgs/model_configs/btcdet_kitti_car.yaml --gpu_str "0" --batch_size 2 --output_dir ../output/kitti_car/ --ckpt_dir  ../output/kitti_car/ckpt/

Multi gpu testing for all saved checkpoints, assuming you have 4 gpus:

bash scripts/dist_test.sh 4 --eval-all --cfg_file ./cfgs/model_configs/btcdet_kitti_car.yaml --gpu_str "0,1,2,3" --batch_size 8 --output_dir ../output/kitti_car/ --ckpt_dir  ../output/kitti_car/ckpt/

Multi gpu testing a specific checkpoint, assuming you have 4 gpus and checkpoint_39 is your best checkpoint :

bash scripts/dist_test.sh 4  --cfg_file ./cfgs/model_configs/btcdet_kitti_car.yaml --gpu_str "0,1,2,3" --batch_size 8 --output_dir ../output/kitti_car/ --ckpt  ../output/kitti_car/ckpt/checkpoint_epoch_39.pth

Run viz:

run this under folder ${dir}/tools/visual_utils

python visualize_pc.py --pc ../../output/kitti_car/eval/eval_all_default/default/cloud/{npy filename}.npy

Jonathan-Twz/BtcDet-benchmark