/3DSFLabelling

This repository is the latest model version corresponding to the paper 3DSFLabeling: Boosting 3D Scene Flow Estimation by Pseudo Auto Labeling.

Primary LanguagePythonMIT LicenseMIT

Celebration

CVPR 2024 | 3DSFLabelling: Boosting 3D Scene Flow Estimation by Pseudo Auto-labelling

CVPR Arxiv

Project Page

GitHub stars GitHub contributors GitHub issues GitHub release (custom) Downloads

Check out the project demo here: 3DSFLabelling-Page

The code is gradually being released, please be patient.

[poster coming soon] [video coming soon]

Description Simple Example of the Auto-Labelling
The proposed 3D scene flow pseudo-auto-labelling framework. Given point clouds and initial bounding boxes, both global and local motion parameters are iteratively optimized. Diverse motion patterns are augmented by randomly adjusting these motion parameters, thereby creating a diverse and realistic set of motion labels for the training of flow estimation models. The proposed 3D scene flow pseudo-auto-labelling framework. Given point clouds and initial bounding boxes, both global and local motion parameters are iteratively optimized. Diverse motion patterns are augmented by randomly adjusting these motion parameters.

Highlights

🔥 GENERATE FULLY ALIGNED INTER-FRAME POINT CLOUDS

We propose a new framework for the automatic labelling of 3D scene flow pseudo-labels, significantly enhancing the accuracy of current scene flow estimation models, and effectively addressing the scarcity of 3D flow labels in autonomous driving.

:fire

🌟 Plug-and-play & Novel motion augmentation

We propose a universal 3D box optimization method with multiple motion attributes. Building upon this, we further introduce a plug-and-play 3D scene flow augmentation module with global-local motions and motion status. This allows for flexible motion adjustment of ego-motion and dynamic environments, setting a new benchmark for scene flow data augmentation.

News

  • [2024/4] 🔥 Open source example model (GMSF, MSBRN and FLOT), Training and evaluation code.
  • [2024/4] 🔥 Open sourced a new data set with 3D scene flow GT, The data comes from KITTI Odometry Dataset, Argoverse Dataset and nuScenes Dataset respectively.
  • [2024/4] 🔥 Open sourced the code for the 3D Scene Flow Label Generation
  • [2024/03] 3DSFLabelling code and models initially released.
  • [2024/02] 3DSFLabelling is accepted by CVPR 2024.
  • [2024/02] 3DSFLabelling paper released.

TODO List

Still in progress:

  • Datasets are easier to use.
  • The validity of the generated labels is verified on motion segmentation and LiDAR odometry.
  • Readability optimization of configuration files and data reading code sections.

Table of Contents

  1. Results and Model Zoo
  2. License and Citation
  3. Comparative Results

Results and Model Zoo

Method Dataset Pre-trained Model EPE3D
GMSF lidarKITTI gmsf_lidarKITTI_epe0.008.pth 0.008
GMSF Argoverse gmsf_argoverse_epe0.013.pth 0.013
GMSF nuScenes gmsf_nuScene_epe0.018.pth 0.018
FLOT lidarKITTI flot_lidarKITTI_epe0.018.tar 0.018
FLOT Argoverse flot_argoverse_epe0.043.tar 0.043
FLOT nuScenes flot_nuScenes_epe0.061.tar 0.061
MSBRN lidarKITTI msbrn_lidarKITTI_epe0.011.pth 0.0110
MSBRN Argoverse msbrn_argoverse_epe0.017.pth 0.017
MSBRN nuScenes msbrn_nuScenes_epe0.076.pth 0.076

Comparative results

The comparative results between our method and baseline. "↑" signifies accuracy enhancement. In real-world LiDAR scenarios, our method markedly improves the 3D flow estimation accuracy across three datasets on the three baselines. This demonstrates that the proposed pseudo-auto-labelling framework can substantially boost the accuracy of existing methods, even without the need for ground truth.

Dataset Method EPE3D↓ Acc3DS↑ Acc3DR↑
FLOT [1] 0.6532 0.1554 0.3130
FLOT+3DSFlabelling 0.0189 ↑97.1% 0.9666 0.9792
MSBRN [2] 0.0139 0.9752 0.9847
LiDAR
KITTI
 MSBRN+3DSFlabelling 0.0123 ↑11.5% 0.9797 0.9868
GMSF [3] 0.1900 0.2962 0.5502
GMSF+3DSFlabelling 0.0078 ↑95.8% 0.9924 0.9947
Dataset Method EPE3D↓ Acc3DS↑ Acc3DR↑
FLOT [1] 0.2491 0.0946 0.3126
FLOT+3DSFlabelling 0.0107 ↑95.7% 0.9711 0.9862
Argoverse MSBRN [2] 0.8691 0.2432 0.2854
MSBRN+3DSFlabelling 0.0150 ↑98.3% 0.9482 0.9601
GMSF [3] 7.2776 0.0036 0.0144
GMSF+3DSFlabelling 0.0093 ↑99.9% 0.9780 0.9880
Dataset Method EPE3D↓ Acc3DS↑ Acc3DR↑
FLOT [1] 0.4858 0.0821 0.2669
FLOT+3DSFlabelling 0.0554 ↑88.6% 0.7601 0.8909
nuScenes MSBRN [2] 0.6137 0.2354 0.2924
MSBRN+3DSFlabelling 0.0235 ↑96.2% 0.9413 0.9604
GMSF [3] 9.4231 0.0034 0.0086
GMSF+3DSFlabelling 0.0185 ↑99.8% 0.9534 0.9713

[1] Puy G, Boulch A, Marlet R. Flot: Scene flow on point clouds guided by optimal transport[C]//European conference on computer vision. Cham: Springer International Publishing, 2020: 527-544.
[2] Cheng W, Ko J H. Multi-scale bidirectional recurrent network with hybrid correlation for point cloud based scene flow estimation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 10041-10050.
[3] Zhang Y, Edstedt J, Wandt B, et al. Gmsf: Global matching scene flow[J]. Advances in Neural Information Processing Systems, 2024, 36.

License and Citation

All assets and code are under the Apache 2.0 license unless specified otherwise.

If this work is helpful for your research, please consider citing the following BibTeX entry.

@inproceedings{yang2023vidar,
  title={3DSFLabelling: Boosting 3D Scene Flow Estimation by Pseudo Auto-labelling},
  author={Jiang, Chaokang and Wang, Guangming and Liu, Jiuming and Wang, Hesheng and Ma, Zhuang and Liu, Zhenqiang and Liang, Zhujin and Shan, Yi and Du, Dalong},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2024}
}