Sparse4D

【2024/06】Our follow-up project, SparseDrive (an end-to-end planning model based on the sparse framework), has been released!!! arxiv & github.

【2024/01/22】The code repository for Sparse4D has been migrated to HorizonRobotics/Sparse4D, and the code for Sparse4Dv3 has also been open-sourced in the new code repository!!! Everyone is welcome to use it.

【2023/11/21】 The paper of Sparse4Dv3 has been published.

Sparse4Dv3 is about to be released, featuring stronger detection performance and end-to-end tracking capabilities.

State-of-the-Art Performance of Sparse4Dv2 in the nuScenes Benchmark for Online Models.

Overall Architecture

Sparse4D v2

Sparse4D v1

nuScenes Benchmark

Validation

These experiments were conducted using 8 RTX 3090 GPUs with 24 GB memory.

model	backbone	pretrain	img size	Epoch	Traning	FPS	NDS	mAP	config	ckpt	log
Sparse4D-T4	Res101	FCOS3D	640x1600	24	2Day5H	2.9	0.5438	0.4409	cfg	ckpt	log
Sparse4Dv2	Res50	ImageNet	256x704	100	15H	20.3	0.5384	0.4392	cfg	ckpt	log
Sparse4Dv2	Res101	nuImage	512x1408	100	2Day	8.4	0.5939	0.5051	-	-	-

Test

model	backbone	img size	NDS	mAP	mATE	mASE	mAOE	mAVE	mAAE
Sparse4D-T4	Vov-99	640x1600	0.595	0.511	0.533	0.263	0.369	0.317	0.124
Sparse4Dv2	Vov-99	640x1600	0.638	0.556	0.462	0.238	0.328	0.264	0.115

Quick Start

Install requirements.

pip install -r requirements.txt
cd projects/mmdet3d_plugin/ops
python setup.py develop

Download nuScenes dataset, pretrain checkpoint(fcos3d.pth ResNet101), pkl files(nuscenes_infos_trainval_with_inds.pkl) and init anchor centers(nuscenes_kmeans900.npy). Adjust the directory structure as follows:

Sparse4D
├── data
│   ├── nuscenes
│   │   ├── maps
│   │   ├── lidarseg
│   │   ├── samples
│   │   ├── sweeps
│   │   ├── v1.0-mini
│   │   ├── v1.0-test
|   |   └── v1.0-trainval
│   ├── nuscenes_cam
│   │   ├── nuscenes_infos_test.pkl
│   │   ├── nuscenes_infos_train.pkl
│   │   ├── nuscenes_infos_val.pkl
│   │   └── nuscenes_infos_trainval_with_inds.pkl
├── projects
│   ├── configs
│   │   ├── default_runtime.py
│   │   ├── sparse4d_r101_H1.py
│   │   ├── sparse4d_r101_H4.py
│   │   └── ...
│   └── mmdet3d_plugin
│       ├── apis
│       ├── core
│       ├── datasets
│       └── models
├── tools
│   ├── dist_test.sh
│   ├── dist_train.sh
│   ├── test.py
│   └── train.py
├── local_test.sh
├── local_train.sh
├── fcos3d.pth
└── nuscenes_kmeans900.npy

Train with config_name.py.

bash local_train.sh config_name

Test checkpoint_file with config_name.py.

bash local_test.sh config_name checkpoint_file

Citation

@misc{2305.14018,
Author = {Xuewu Lin and Tianwei Lin and Zixiang Pei and Lichao Huang and Zhizhong Su},
Title = {Sparse4D v2: Recurrent Temporal Fusion with Sparse Model},
Year = {2023},
Eprint = {arXiv:2305.14018},
}

@misc{2211.10581,
  Author = {Xuewu Lin and Tianwei Lin and Zixiang Pei and Lichao Huang and Zhizhong Su},
  Title = {Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusion},
  Year = {2022},
  Eprint = {arXiv:2211.10581},
}

linxuewu/Sparse4D