Language Prompt for Autonomous Driving

Language Prompt for Autonomous Driving

Dongming Wu*, Wencheng Han*, Tiancai Wang, Yingfei Liu, Xiangyu Zhang, Jianbing Shen

🔥 Introduction

This is the official implementation of Language Prompt for Autonomous Driving.

We propose a new large-scale language prompt set for driving scenes, named NuPrompt. As far as we know, it is the first dataset specializing in multiple 3D objects of interest from video domain.
We construct a new prompt-based driving perceiving task, which requires using a language prompt as a semantic cue to predict object trajectories.
We develop a simple end-to-end baseline model, called PromptTrack, which effectively fuses cross-modal features in a newly built prompt reasoning branch to predict referent objects, showing impressive performance.

💥 News

[2024.06.27] Data and code are released. Welcome to try it!
[2023.09.11] Our paper is released at arXiv.

⭐ Benchmark

We expand nuScenes dataset with annotating language prompts, named NuPrompt. It is a large-scale dataset for language prompt in driving scenes, which contains 40,147 language prompts for 3D objects. Thanks to nuScenes, our descriptions are closed to real-driving nature and complexity, covering a 3D, multi-view, and multi-frame space.

The data can be downloaded from NuPrompt.

🔨 Model

Our model is built upon PF-Track.

Please refer to data.md for preparing data and pre-trained models.

Please refer to environment.md for environment installation.

Please refer to training_inference.md for training and evaluation.

🚀 Results

Method	AMOTA	AMOTP	RECALL	Model	Config
PromptTrack	0.200	1.572	32.5%	model	config

👉 Citation

If you find our work useful in your research, please consider citing them.

@article{wu2023language,
  title={Language Prompt for Autonomous Driving},
  author={Wu, Dongming and Han, Wencheng and Wang, Tiancai and Liu, Yingfei and Zhang, Xiangyu and Shen, Jianbing},
  journal={arXiv preprint},
  year={2023}
}

@inproceedings{wu2023referring,
  title={Referring multi-object tracking},
  author={Wu, Dongming and Han, Wencheng and Wang, Tiancai and Dong, Xingping and Zhang, Xiangyu and Shen, Jianbing},
  booktitle={CVPR},
  year={2023}
}

❤️ Acknowledgements

We thank the authors that open the following projects.

wudongming97/Prompt4Driving