/DUSA

[ACM MM 2023] Official implementation of DUSA: Decoupled Unsupervised Sim2Real Adaptation for Vehicle-to-Everything Collaborative Perception

Primary LanguagePythonOtherNOASSERTION

DUSA: Decoupled Unsupervised Sim2Real Adaptation for Vehicle-to-Everything Collaborative Perception

Official implementation of the ACM MM 2023 paper DUSA: Decoupled Unsupervised Sim2Real Adaptation for Vehicle-to-Everything Collaborative Perception [Link].

News

  • 11/17/2023: Initial release 🎉🎉🎉.

Features

Data Preparation

OPV2V & V2XSet

All the data can be downloaded from google drive: OPV2V, V2XSet. If you have a good Internet connection, you may directly download the complete large zip file such as train.zip. In case you suffer from downloading large files, the link also provides split small chunks, which can be found in the directory ending with _chunks, such as train_chunks. After downloading, please run the following command to each set to merge those chunks together:

cat train.zip.part* > train.zip
unzip train.zip

After extraction, please make the file structured as following:

├── v2xset  # the downloaded data
    ├── train
    ├── validate
    ├── test

DAIR-V2X-C

The DAIR-V2X-C dataset can be downloaded from the official website.

Complemented Annotations for DAIR-V2X-C 🌟

Originally, DAIR-V2X-C only annotates 3D boxes within the range of camera's view in vehicle-side. CoAlign supplement the missing 3D box annotations to enable the 360 degree detection. With fully complemented vehicle-side labels, CoAlign regenerate the cooperative labels for users, which follow the original cooperative label format.

Download: Google Drive

Website: Website

Installation

Please follow the commands in install.sh to install the dependencies. We recommend using conda to manage the environment.

You may also refer to data introduction and installation of OpenCOOD to help you understand the project structure.

Quick Start

Data Sequence Visualization

To quickly visualize the LiDAR stream in the OPV2V dataset, first modify the validate_dir in your opencood/hypes_yaml/visualization.yaml to the OPV2V data path on your local machine, e.g. opv2v/validate, and then run the following command:

cd ~/DUSA
python opencood/visualization/vis_data_sequence.py [--color_mode ${COLOR_RENDERING_MODE}]

Arguments Explanation:

  • color_mode : str type, indicating the lidar color rendering mode. You can choose from 'constant', 'intensity' or 'z-value'.

Training

DUSA uses yaml file to configure most of the parameters for training. To train your own model from scratch or a specific checkpoint, run the following commands:

python opencood/tools/train.py --hypes_yaml ${CONFIG_FILE} [--model_dir  ${CHECKPOINT_FOLDER} --half --adv_training --target_domain_suffix ${TARGET_DOMAIN} --use_pseudo_label --pseudo_label_id ${ID}]

Arguments Explanation:

  • hypes_yaml: the path of the training configuration file, e.g. opencood/hypes_yaml/second_early_fusion.yaml, meaning you want to train an early fusion model which utilizes SECOND as the backbone. See Tutorial 1: Config System to learn more about the rules of the yaml files.
  • model_dir (optional) : the path of the checkpoints. This is used to fine-tune the trained models. When the model_dir is given, the trainer will discard the hypes_yaml and load the config.yaml in the checkpoint folder.
  • half (optional): If set, the model will be trained with half precision. It cannot be set with multi-gpu training togetger.
  • adv_training (optional): If set, the model will be trained with adversarial training.
  • target_domain_suffix (optional): Yaml filename suffix of the target domain of adversarial training. Default: dair.
  • use_pseudo_label (optional): If set, the model will be trained with pseudo label.
  • pseudo_label_id (optional): The id of the pseudo label. Default: 0.

To train on multiple gpus, run the following command:

CUDA_VISIBLE_DEVICES="0,1,2,3" python -m torch.distributed.launch --nproc_per_node=4  --use_env opencood/tools/train.py --hypes_yaml ${CONFIG_FILE} [--model_dir  ${CHECKPOINT_FOLDER} --adv_training --target_domain_suffix ${TARGET_DOMAIN} --use_pseudo_label --pseudo_label_id ${ID}]

Inference

Before you run the following command, first make sure the validation_dir in config.yaml under your checkpoint folder refers to the testing dataset path, e.g. opv2v_data_dumping/test.

python opencood/tools/inference.py --model_dir ${CHECKPOINT_FOLDER} --fusion_method ${FUSION_STRATEGY} [--show_vis] [--show_sequence] [--save_pseudo_label] [--pseudo_label_id ${ID}]

Arguments Explanation:

  • model_dir: the path to your saved model.
  • fusion_method: indicate the fusion strategy, currently support 'early', 'late', and 'intermediate'.
  • show_vis(optional): whether to visualize the detection overlay with point cloud.
  • show_sequence (optional): the detection results will visualized in a video stream. It can NOT be set with show_vis at the same time.
  • save_pseudo_label (optional): whether to save the pseudo label for the target domain.
  • pseudo_label_id (optional): The id of the pseudo label. Default: 0.

The evaluation results will be dumped in the model directory.

Citation

If you are using DUSA for your research, please cite the following paper:

@inproceedings{kong2023dusa,
 title={DUSA: Decoupled Unsupervised Sim2Real Adaptation for Vehicle-to-Everything Collaborative Perception},
 author={Kong, Xianghao and Jiang, Wentao and Jia, Jinrang and Shi, Yifeng and Xu, Runsheng and Liu, Si},
 booktitle={Proceedings of the 31st ACM International Conference on Multimedia},
 pages={1943--1954},
 year={2023}
}

Acknowledgements

This project is built upon OpenCOOD and CoAlign. Thanks again to @DerrickXuNu for his great code framework.