This is a repo of InterFusion for 3D object detection.
The code is mainly based on OpenPCDet.
Many recent works detect 3D objects by several sensor modalities for autonomous driving, where high-resolution cameras and high-line Lidars are mostly used but relatively expensive. To achieve a balance between overall cost and detection accuracy, many multi-modal fusion techniques have been suggested. In recent years, the fusion of Lidar and Radar has gained ever-increasing attention, especially 4D Radar, which can adapt to bad weather conditions due to its penetrability. Although features have been fused from multiple sensing modalities, most methods cannot learn interactions from different modalities, which does not make for their best use. Inspired by the self-attention mechanism, here we present InterFusion, an interaction-based fusion framework, to fuse 16-line Lidar with 4D Radar. It aggregates features from two modalities and identifies cross-modal relations between Radar and Lidar features. In experimental evaluations on the Astyx HiRes2019 dataset, our method outperformed the baseline by 4.09% mAP in 3D and 10.34% BEV mAP for the car class at the moderate level.
- Model Framework:
- All experiments are tested on Astyx Hires2019
| Modality | Method 3D | mAP(%) BEV mAP(%) | ||||
| Radar | Easy | Moderate | Hard | Easy | Moderate | Hard |
| PointRCNN | 12.23 | 9.1 | 9.1 | 14.95 | 13.82 | 13.89 |
| SECOND [20] | 24.11 | 18.5 | 17.77 | 41.25 | 30.58 | 29.33 |
| PVRCNN [33] | 28.21 | 22.29 | 20.4 | 46.62 | 35.1 | 33.67 |
| PointPillars [30] | 30.14 | 24.06 | 21.91 | 45.66 | 36.71 | 35.3 |
| Lidar | ||||||
| PointRCNN [23] | 30.67 | 23.69 | 23.03 | 35.75 | 28.13 | 23.79 |
| SECOND [20] | 53.32 | 44.1 | 40.16 | 57.26 | 47.52 | 45.4 |
| PVRCNN [33] | 54.93 | 45.29 | 41.4 | 56.71 | 47.55 | 45.06 |
| PointPillars [30] | 53.02 | 43.56 | 41.72 | 55.76 | 45.81 | 43.62 |
| Radar+Lidar Ours(InterFusion) | 59.04 | 47.65 | 46.47 | 68.1 | 56.15 | 55.01 |
| Delta | +6.02 | +4.09 | +4.75 | +12.34 | +10.34 | +11.39 |
a. Clone this repository.
git clone https://github.com/adept-thu/InterFusion.gitb. Install the dependent libraries as follows:
- Install the dependent python libraries:
pip install -r requirements.txt
c. Generate dataloader
python -m pcdet.datasets.astyx.astyx_dataset create_astyx_infos tools/cfgs/dataset_configs/astyx_dataset.yaml
CUDA_VISIBLE_DEVICES=1 python train.py --cfg_file cfgs/astyx_models/pointpillar.yaml --tcp_port 25851 --extra_tag yourmodelname
python test.py --cfg_file cfgs/astyx_models/pointpillar.yaml --batch_size 4 --ckpt ##astyx_models/pointpillar/debug/ckpt/checkpoint_epoch_80.pth
If you find this project useful in your research, please consider cite: