Li-Chengyang/2DPASS

2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds (ECCV 2022) :fire:

2DPASS

This repository is for 2DPASS introduced in the following paper

Xu Yan*, Jiantao Gao*, Chaoda Zheng*, Chao Zheng, Ruimao Zhang, Shuguang Cui, Zhen Li*, "2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds", ECCV 2022 [arxiv].

If you find our work useful in your research, please consider citing:

@inproceedings{yan20222dpass,
  title={2dpass: 2d priors assisted semantic segmentation on lidar point clouds},
  author={Yan, Xu and Gao, Jiantao and Zheng, Chaoda and Zheng, Chao and Zhang, Ruimao and Cui, Shuguang and Li, Zhen},
  booktitle={European Conference on Computer Vision},
  pages={677--695},
  year={2022},
  organization={Springer}
}

@InProceedings{yan2022let,
      title={Let Images Give You More: Point Cloud Cross-Modal Training for Shape Analysis}, 
      author={Xu Yan and Heshen Zhan and Chaoda Zheng and Jiantao Gao and Ruimao Zhang and Shuguang Cui and Zhen Li},
      year={2022},
      booktitle={NeurIPS}
}

News

• 2022-10-11 Our new work for cross-modal knowledge distillation is accepted at NeurIPS 2022:smiley: (here).
• 2022-09-20 We release codes for SemanticKITTI single-scan and NuScenes 🚀!
• 2022-07-03 2DPASS is accepted at ECCV 2022 🔥!
• 2022-03-08 We achieve 1st place in both single and multi-scans of SemanticKITTI and 3rd place on NuScenes-lidarseg 🔥!

Installation

Requirements

• pytorch >= 1.8
• yaml
• easydict
• pyquaternion
• lightning (tested with pytorch_lightning==1.3.8 and torchmetrics==0.5)
• torch-scatter (pip install torch-scatter -f https://data.pyg.org/whl/torch-1.9.0+${CUDA}.html)
• nuScenes-devkit (optional for nuScenes)
• spconv (tested with spconv==2.1.16 and cuda==11.1, pip install spconv-cu111==2.1.16)

Data Preparation

SemanticKITTI

Please download the files from the SemanticKITTI website and additionally the color data from the Kitti Odometry website. Extract everything into the same folder.

./dataset/
├── 
├── ...
└── SemanticKitti/
    ├──sequences
        ├── 00/           
        │   ├── velodyne/	
        |   |	├── 000000.bin
        |   |	├── 000001.bin
        |   |	└── ...
        │   └── labels/ 
        |   |   ├── 000000.label
        |   |   ├── 000001.label
        |   |   └── ...
        |   └── image_2/ 
        |   |   ├── 000000.png
        |   |   ├── 000001.png
        |   |   └── ...
        |   calib.txt
        ├── 08/ # for validation
        ├── 11/ # 11-21 for testing
        └── 21/
	    └── ...

NuScenes

Please download the Full dataset (v1.0) from the NuScenes website with lidarseg and extract it.

./dataset/
├── 
├── ...
└── nuscenes/
    ├──v1.0-trainval
    ├──v1.0-test
    ├──samples
    ├──sweeps
    ├──maps
    ├──lidarseg

Training

SemanticKITTI

You can run the training with

cd <root dir of this repo>
python main.py --log_dir 2DPASS_semkitti --config config/2DPASS-semantickitti.yaml --gpu 0

The output will be written to logs/SemanticKITTI/2DPASS_semkitti by default.

NuScenes

cd <root dir of this repo>
python main.py --log_dir 2DPASS_nusc --config config/2DPASS-nuscenese.yaml --gpu 0 1 2 3

Vanilla Training without 2DPASS

We take SemanticKITTI as an example.

cd <root dir of this repo>
python main.py --log_dir baseline_semkitti --config config/2DPASS-semantickitti.yaml --gpu 0 --baseline_only

Testing

You can run the testing with

cd <root dir of this repo>
python main.py --config config/2DPASS-semantickitti.yaml --gpu 0 --test --num_vote 12 --checkpoint <dir for the pytorch checkpoint>

Here, num_vote is the number of views for the test-time-augmentation (TTA). We set this value to 12 as default (on a Tesla-V100 GPU), and if you use other GPUs with smaller memory, you can choose a smaller value. num_vote=1 denotes there is no TTA used, and will cause about ~2% performance drop.

Model Zoo

You can download the models with the scores below from this Google drive folder.

SemanticKITTI

Model (validation)	mIoU (vanilla)	mIoU (TTA)	Parameters
2DPASS (4scale-64dimension)	68.7%	70.0%	1.9M

Here, we fine-tune the model on SemanticKITTI with more epochs and thus gain the higher mIoU. If you train with 64 epochs, it should be gained about 66% for vanilla and 69% after TTA.

NuScenes

Model (validation)	mIoU (vanilla)	mIoU (TTA)	Parameters
2DPASS (6scale-128dimension)	76.7%	79.6%	11.5M
2DPASS (6scale-256dimension)	78.0%	80.5%	45.6M

Note that the results on benchmarks are gained by training with additional validation set and using instance-level augmentation.

Acknowledgements

Code is built based on SPVNAS, Cylinder3D, xMUDA and SPCONV.

License

This repository is released under MIT License (see LICENSE file for details).