Official implementation of the ACM MM 2023 paper DUSA: Decoupled Unsupervised Sim2Real Adaptation for Vehicle-to-Everything Collaborative Perception [Link].
- 11/17/2023: Initial release 🎉🎉🎉.
-
Provide easy data API for multiple popular multi-agent perception dataset
-
Provide multiple SOTA 3D detection backbone
-
Support multiple sparse convolution versions
- Spconv 1.2.1
- Spconv 2.x
-
Support SOTA multi-agent perception models
-
Provide a convenient log replay toolbox for the OPV2V dataset. Check here for more details.
All the data can be downloaded from google drive: OPV2V, V2XSet. If you have a good Internet connection, you may directly download the complete large zip file such as train.zip
. In case you suffer from downloading large files, the link also provides split small chunks, which can be found in the directory ending with _chunks
, such as train_chunks
. After downloading, please run the following command to each set to merge those chunks together:
cat train.zip.part* > train.zip
unzip train.zip
After extraction, please make the file structured as following:
├── v2xset # the downloaded data
├── train
├── validate
├── test
The DAIR-V2X-C dataset can be downloaded from the official website.
Originally, DAIR-V2X-C only annotates 3D boxes within the range of camera's view in vehicle-side. CoAlign supplement the missing 3D box annotations to enable the 360 degree detection. With fully complemented vehicle-side labels, CoAlign regenerate the cooperative labels for users, which follow the original cooperative label format.
Download: Google Drive
Website: Website
Please follow the commands in install.sh
to install the dependencies. We recommend using conda to manage the environment.
You may also refer to data introduction and installation of OpenCOOD to help you understand the project structure.
To quickly visualize the LiDAR stream in the OPV2V dataset, first modify the validate_dir
in your opencood/hypes_yaml/visualization.yaml
to the OPV2V data path on your local machine, e.g. opv2v/validate
, and then run the following command:
cd ~/DUSA
python opencood/visualization/vis_data_sequence.py [--color_mode ${COLOR_RENDERING_MODE}]
Arguments Explanation:
color_mode
: str type, indicating the lidar color rendering mode. You can choose from 'constant', 'intensity' or 'z-value'.
DUSA uses yaml file to configure most of the parameters for training. To train your own model from scratch or a specific checkpoint, run the following commands:
python opencood/tools/train.py --hypes_yaml ${CONFIG_FILE} [--model_dir ${CHECKPOINT_FOLDER} --half --adv_training --target_domain_suffix ${TARGET_DOMAIN} --use_pseudo_label --pseudo_label_id ${ID}]
Arguments Explanation:
hypes_yaml
: the path of the training configuration file, e.g.opencood/hypes_yaml/second_early_fusion.yaml
, meaning you want to train an early fusion model which utilizes SECOND as the backbone. See Tutorial 1: Config System to learn more about the rules of the yaml files.model_dir
(optional) : the path of the checkpoints. This is used to fine-tune the trained models. When themodel_dir
is given, the trainer will discard thehypes_yaml
and load theconfig.yaml
in the checkpoint folder.half
(optional): If set, the model will be trained with half precision. It cannot be set with multi-gpu training togetger.adv_training
(optional): If set, the model will be trained with adversarial training.target_domain_suffix
(optional): Yaml filename suffix of the target domain of adversarial training. Default:dair
.use_pseudo_label
(optional): If set, the model will be trained with pseudo label.pseudo_label_id
(optional): The id of the pseudo label. Default:0
.
To train on multiple gpus, run the following command:
CUDA_VISIBLE_DEVICES="0,1,2,3" python -m torch.distributed.launch --nproc_per_node=4 --use_env opencood/tools/train.py --hypes_yaml ${CONFIG_FILE} [--model_dir ${CHECKPOINT_FOLDER} --adv_training --target_domain_suffix ${TARGET_DOMAIN} --use_pseudo_label --pseudo_label_id ${ID}]
Before you run the following command, first make sure the validation_dir
in config.yaml under your checkpoint folder refers to the testing dataset path, e.g. opv2v_data_dumping/test
.
python opencood/tools/inference.py --model_dir ${CHECKPOINT_FOLDER} --fusion_method ${FUSION_STRATEGY} [--show_vis] [--show_sequence] [--save_pseudo_label] [--pseudo_label_id ${ID}]
Arguments Explanation:
model_dir
: the path to your saved model.fusion_method
: indicate the fusion strategy, currently support 'early', 'late', and 'intermediate'.show_vis
(optional): whether to visualize the detection overlay with point cloud.show_sequence
(optional): the detection results will visualized in a video stream. It can NOT be set withshow_vis
at the same time.save_pseudo_label
(optional): whether to save the pseudo label for the target domain.pseudo_label_id
(optional): The id of the pseudo label. Default:0
.
The evaluation results will be dumped in the model directory.
If you are using DUSA for your research, please cite the following paper:
@inproceedings{kong2023dusa,
title={DUSA: Decoupled Unsupervised Sim2Real Adaptation for Vehicle-to-Everything Collaborative Perception},
author={Kong, Xianghao and Jiang, Wentao and Jia, Jinrang and Shi, Yifeng and Xu, Runsheng and Liu, Si},
booktitle={Proceedings of the 31st ACM International Conference on Multimedia},
pages={1943--1954},
year={2023}
}
This project is built upon OpenCOOD and CoAlign. Thanks again to @DerrickXuNu for his great code framework.