Towards Stable 3D Object Detection (ECCV 2024)

Jiabao Wang*, Qiang Meng*, Guochao Liu, Liujiang Yan, Ke Wang, Ming-Ming Cheng, Qibin Hou#

Project / Paper

Abstract

In autonomous driving, the temporal stability of 3D object detection greatly impacts the driving safety. However, the detection stability cannot be accessed by existing metrics such as mAP and MOTA, and consequently is less explored by the community. To bridge this gap, this work proposes Stability Index (SI), a new metric that can comprehensively evaluate the stability of 3D detectors in terms of confidence, box localization, extent, and heading. By benchmarking state-of-the-art object detectors on the Waymo Open Dataset, SI reveals interesting properties of object stability that have not been previously discovered by other metrics. To help models improve their stability, we further introduce a general and effective training strategy, called Prediction Consistency Learning (PCL). PCL essentially encourages the prediction consistency of the same objects under different timestamps and augmentations, leading to enhanced detection stability. Furthermore, we examine the effectiveness of PCL with the widely-used CenterPoint, and achieve a remarkable SI of 86.00 for vehicle class, surpassing the baseline by 5.48. We hope our work could serve as a reliable baseline and draw the community's attention to this crucial issue in 3D object detection.

Demos

Confidence	Localization

Extent	Heading

Main Results

We could not provide pre-trained models due to Waymo Dataset License Agreement. However, one can easily reproduce similar performances with our provided default configurations.

Benckmark on Waymo Open Dataset

Methods	Vehicle(%)						Pedestrain(%)						Cyclist(%)
Methods	mAPH_l1	SI	SI_c	SI_l	SI_e	SI_h	mAPH_l1	SI	SI_c	SI_l	SI_e	SI_h	mAPH_l1	SI	SI_c	SI_l	SI_e	SI_h
Second	72.60	81.32	90.17	84.18	91.95	92.10	59.81	62.80	83.94	69.34	87.30	67.31	61.95	66.29	81.94	75.36	85.99	81.29
CenterPoint-Pillar	72.44	80.61	89.03	85.37	91.00	92.82	65.59	64.57	83.24	74.43	87.38	68.85	67.36	68.06	80.77%	77.66	86.96	85.89
PointPillar	72.84	80.84	89.58	84.40	92.26	91.60	54.64	62.03	84.65	72.08	88.83	57.93	59.51	66.14	82.15%	74.85	88.00	77.36
CenterPoint	73.73	80.52	89.04	85.33	90.70	92.88	69.50	68.40	85.74	73.28	88.58	74.95	71.04	68.40	80.31%	78.47	87.37	89.79
PartA2Net	75.02	82.86	91.43	85.38	91.68	91.69	66.16	65.08	84.59	73.61	86.71	67.04	67.90	72.73	85.94%	79.34	86.97	84.29
PV R-CNN	75.92	83.73	91.94	86.36	92.30	91.66	66.28	66.17	86.02	73.50	87.39	66.58	68.38	73.53	86.84%	78.86	88.44	83.22
Voxel R-CNN	77.19	84.26	92.01	86.66	92.11	93.33	74.21	69.50	86.87	75.33	88.06	73.56	71.68	73.23	84.42%	80.07	87.67	89.28
VoxelNeXt	77.84	84.82	92.88	86.28	91.59	94.17	76.24	74.74	92.67	75.65	88.03	75.80	75.59	76.48	89.98%	79.24	84.90	87.76
PV R-CNN++	77.88	84.49	92.06	87.22	92.38	93.18	73.99	69.27	86.75	75.27	88.09	73.24	71.84	73.05	84.23%	80.29	87.66	89.19
PV R-CNN++ Res	78.33	85.17	92.47	87.52	92.51	93.85	75.75	70.15	87.16	75.83	87.89	74.81	72.47	73.31	84.25%	80.61	87.58	89.65
DSVT	78.82	84.90	92.51	86.90	91.53	94.76	76.81	74.58	91.88	76.47	88.71	75.92	75.44	76.20	88.22%	80.48	86.11	89.88
TransFusion	79.00	82.32	89.34	86.8	92.73	95.68	76.52	69.11	84.53	75.39	89.89	78.78	70.11	70.35	80.63%	79.49	90.56	91.13

Analysis on Waymo Open Dataset

Results of PCL

Methods	Vehicle(%)		Pedestrian(%)		Cyclist(%)
Methods	mAPH_l1	SI	mAPH_l1	SI	mAPH_l1	SI
Baseline	73.73	80.52	69.50	68.40	71.04	68.40
w/o PCL	73.70	80.93	69.55	68.35	71.27	68.20
PCL (n=0)	75.57	85.42	70.18	71.87	70.86	68.80
PCL (n=4)	75.26	85.83	69.56	72.76	70.65	69.22
PCL (n=8)	75.04	85.94	68.82	72.87	70.31	69.32
PCL (n=12)	74.64	85.93	68.50	72.95	70.85	69.33
PCL (n=16)	74.54	86.00	67.82	73.14	70.25	69.16

Installation

This repository is implemented on top of OpenPCDet. For package installation and data preparation, please follow the guidance of the INSTALL.md and GETTING_STARTED.md in OpenPCDet.

Training and Evaluation

All our experimental configurations can be found in waymo_stable. Steps to reproduce results in our paper include:

Training Baseline CenterPoint

The first step is to train a baseline CenterPoint model configured with centerpoint_baseline.yaml.

cd tools
bash scripts/dist_train.sh 8 --cfg_file cfgs/waymo_stable/centerpoint_baseline.yaml

Fine-tuning with PCL

We provide the configurations of fine-tuning CenterPoint without PCL (centerpoint_finetune.yaml) and with PCL (centerpoint_PCL_n{X}.yaml). Here, X is the maximum interval between the neighborhood sampling, as described in our paper.

Please do not forget to fill in the correct path to baseline checkpoint in argument PRETRAINED before the fine-tuning step.

MODEL:
    NAME: CenterPoint
    # change this path to the baseline checkpoint.
    PRETRAINED: '../output/waymo_stable/centerpoint_baseline/default/ckpt/checkpoint_epoch_36.pth'

Then, you can fine-tune the CenterPoint model with or without our PCL strategy.

Fine-tune without PCL:

cd tools
bash scripts/dist_train.sh 8 --cfg_file cfgs/waymo_stable/centerpoint_finetune.yaml

Fine-tune with PCL

cd tools
bash scripts/dist_train.sh 8 --cfg_file cfgs/waymo_stable/centerpoint_PCL_n16.yaml

Evaluation with Stability Index

Stability index (SI) has been integrated in waymo_dataset.py and nuscene_dataset.py. One can enable SI evaluation by simply adding 'stability_index' to POST_PROCESSING.EVAL_METRIC in the configuration, as exemplified by EVAL_METRIC in centerpoint_PCL_n16.yaml.

POST_PROCESSING:
    RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
    EVAL_METRIC: ['waymo', 'stable_index']

Then, the mAPH result can be achieved by performing:

cd tools
python test.py \
    --cfg_file cfgs/waymo_stable/centerpoint_PCL_n16.yaml \
    --ckpt ../output/waymo_stable/centerpoint_PCL_n16/default/ckpt/checkpoint_epoch_5.pth

In the end, the SI can be evaluated by the following commands along with the generated result.pkl:

python pcdet/datasets/waymo/waymo_stability_index.py \
    --gt_infos data/waymo/waymo_processed_data_v0_5_0_infos_val.pkl \
    --pred_infos output/waymo_stable/centerpoint_PCL_n16/default/eval/eval_with_train/epoch_5/result.pkl

Acknowledgement

Our code is developed on top of the open source codebase OpenPCDet. We sincerely appreciate their amazing works.

jbwang1997/StabilityIndex