Awesome Radar-Camera Fusion

Overview

Surveys

  • 2023 - Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review TIV [Paper] [Website] [GitHub]
  • 2021 - Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges TITS [Paper] [Website]
  • 2023 - Vision-RADAR fusion for Robotics BEV Detections: A Survey IV [Paper]

Datasets

Id Name Year Task Annotation Radar Data Representation Link
1 nuScenes 2019 Object Detection
Object Tracking
3D box Point Cloud Website Github
2 Astyx 2019 Object Detection 3D box Point Cloud Website
3 SeeingThroughFog 2020 Object Detection 2D box
3D box
Point Cloud Website
4 CARRADA 2020 Object Detection
Semantic Segmentation
Object Tracking
Trajectory Prediction
2D box
2D pixel
Range-Doppler Tensor
Range-Azimuth Tensor
Website
5 HawkEye 2020 Semantic Segmentation 3D point Point Cloud Website
6 Zendar 2020 Object Detection
Mapping
Localization
2D box Range-Doppler Tensor
Range-Azimuth Tensor
Point Cloud
Website
7 RADIATE 2020 Object Detection
Object Tracking
SLAM
Scene Understanding
2D box Range-Azimuth Tensor Website
8 AIODrive 2020 Object Detection
Object Tracking
Semantic Segmentation
Trajectory Prediction
Depth Estimation
2D box
3D box
Point Cloud Website
9 CRUW 2021 Object Detection 2D box Range-Azimuth Tensor Website
10 RaDICaL 2021 Object Detection 2D box ADC Signal Website
11 RadarScenes 2021 Object Detection
Semantic Segmentation
Object Tracking
2D pixel
3D point
Point Cloud Website
12 RADDet 2021 Object Detection 2D box
3D box
Range-Azimuth-Doppler Tensor Github
13 FloW 2021 Object Detection 2D box Range-Doppler Tensor
Point Cloud
Website Github
14 RADIal 2021 Object Detection
Semantic Segmentation
2D box ADC Signal
Range-Azimuth-Doppler Tensor
Range-Azimuth Tensor
Range-Doppler Tensor
Point Cloud
Github
15 VoD 2022 Object Detection 2D box
3D box
4D Point Cloud Website
16 Boreas 2022 Object Detection
Localization
Odometry
2D box Range-Azimuth Tensor Website
17 TJ4DRadSet 2022 Object Detection
Object Tracking
3D box 4D Point Cloud Website
18 K-Radar 2022 Object Detection
Object Tracking
SLAM
3D box 4D Range-Azimuth-Doppler Tensor Github
19 aiMotive 2022 Object Detection 3D box Point cloud Website
20 WaterScenes 2023 Instance Segmentation
Semantic Segmentation
Free-space Segmentation
Waterline Segmentation
Panoptic Perception
2D box
2D pixel
2D line
3D point
4D Point cloud Paper Website GitHub
21 ThermRad 2023 Object Detection 3D box 4D Point Cloud Paper
22 Dual Radar 2023 Object Detection
Object Tracking
3D box 4D Point Cloud Paper GitHub

Methods

Id Short Name Year Task Annotation Radar Data Representation Fusion Level Dataset Link
1 Chadwick et al. 2019 Object Detection 2D box Point Cloud Feature Level Self-Recorded
2 RRPN 2019 Object Detection 2D box Point Cloud Data Level nuScenes Code
3 Jha et al. 2019 Object Detection 2D box Point Cloud Object Level Self-Recorded
4 CMGGAN 2019 Semantic Segmentation 2D point Grid Map Feature Level Self-Recorded
5 Meyer and Kuschk 2019 Object Detection 3D box Point Cloud Data Level Astyx
6 RVNet 2019 Object Detection 2D box Point Cloud Feature Level nuScenes
7 FusionNet 2019 Object Detection
Object Classification
2D box Range-Azimuth Tensor Feature Level Self-Recorded
8 SO-Net 2020 Object Detection
Semantic Segmentation
2D box
2D pixel
Point Cloud Feature Level nuScenes
9 SAF-FCOS 2020 Object Detection 2D box Point Cloud Feature Level nuScenes Code
10 CRF-Net 2019 Object Detection 2D box Point Cloud Data Level nuScenes
Self-Recorded
Code
11 Bijelic et al. 2020 Object Detection 2D box Point Cloud Feature Level DENSE Code
12 BIRANet 2020 Object Detection 2D box Point Cloud Feature Level nuScenes Code
13 Nabati and Qi 2020 Object Detection
Depth Estimation
2D box Point Cloud Mixed Level nuScenes
14 YOdar 2020 Object Detection 2D box Point Cloud Feature Level nuScenes
15 CenterNet 2020 Object Detection 3D box Point Cloud Feature Level nuScenes Code
16 RODNet 2020 Object Detection 2D box Range-Azimuth Tensor Feature Level CRUW Code
17 RAMP-CNN 2021 Object Detection 2D box Range-Azimuth-Doppler Tensor Feature Level CRUW
18 Li and Xie 2021 Object Detection 3D box Point Cloud Feature Level nuScenes
19 Kim et al. 2020 Object Detection 3D box Range-Azimuth Tensor Feature Level Self-Recorded
20 AssociationNet 2021 Object Detection 2D box Point Cloud Object Level Self-Recorded
21 RVF-Net 2021 Object Detection 3D box Point Cloud Data Level nuScenes
22 Cui et al. 2021 Object Detection 3D box Point Cloud Mixed Level Astyx
23 RISFNet 2021 Object Detection 2D box Point Cloud Feature Level FloW
24 GRIF Net 2021 Object Detection 3D box Point Cloud Feature Level nuScenes
25 Stacker et al. 2021 Object Detection 2D box Point Cloud Feature Level nuScenes
26 Harley et al. 2021 Semantic Segmentation 2D pixel Point Cloud Feature Level nuScenes
27 RadSegNet 2022 Object Detection 2D box
2D pixel
Point Cloud
Range-Azimuth Tensor
Data Astyx
RADIATE
28 RCBEV 2022 Object Detection 3D box Point Cloud Feature Level nuScenes
29 CRAFT 2022 Object Detection 3D box Point Cloud Data Level nuScenes
30 DeepFusion 2022 Object Detection 3D box Point Cloud Feature Level Self-reorded
nuScenes
31 CramNet 2022 Object Detection 3D box Range-Azimuth Tensor Feature Level RADIATE
32 MVFusion 2023 Object Detection 3D box Point Cloud Feature Level nuScenes
33 CRN 2023 Object Detection 3D box Point Cloud Feature Level nuScenes

Radar-Camera

Detection

  • 2022 - Detecting Darting Out Pedestrians With Occlusion Aware Sensor Fusion of Radar and Stereo Camera TIV [Paper]
  • 2023 - RCFusion: Fusing 4-D Radar and Camera With Bird’s-Eye View Features for 3-D Object Detection [VoD TJ4DRadSet] TIM [Paper]
  • 2023 - LXL: LiDAR Exclusive Lean 3D Object Detection with 4D Imaging Radar and Camera Fusion [VoD TJ4DRadSet] TIV [Paper]
  • 2023 - REDFormer: Radar Enlightens the Darkness of Camera Perception with Transformers [nuScenes] TIV [Paper]
  • 2023 - SparseFusion3D: Sparse Sensor Fusion for 3D object detection by Radar and Camera in Environmental Perception [nuScenes] TIV [Paper]

Segmentation

Tracking

  • 2021 - A Novel Spatio-Temporal Synchronization Method of Roadside Asynchronous MMW Radar-Camera for Sensor Fusion TITS [Paper]
  • 2021 - CFTrack: Center-based Radar and Camera Fusion for 3D Multi-Object Tracking IV Workshops [Paper]
  • 2021 - 3D Detection and Tracking for On-road Vehicles with a Monovision Camera and Dual Low-cost 4D mmWave Radars ITSC [Paper]
  • 2022 - Robust Target Recognition and Tracking of Self-Driving Cars With Radar and Camera Information Fusion Under Severe Weather Conditions TITS [Paper]

Multi-Task

  • 2023 - Achelous: A Fast Unified Water-surface Panoptic Perception Framework based on Fusion of Monocular Camera and 4D mmWave Radar [WaterScenes] ITSC [Paper] [GitHub]
  • 2023 - Mask-VRDet: Mask-VRDet: A robust riverway panoptic perception model based on dual graph fusion of vision and 4D mmWave radar[WaterScenes] RAS [Paper] [GitHub]
  • 2023 - Efficient-VRNet: An Exquisite Fusion Network for Riverway Panoptic Perception based on Asymmetric Fair Fusion of Vision and 4D mmWave Radar [WaterScenes] arXiv [Paper] [GitHub]

Radar-LiDAR

Detection

  • 2022 - RaLiBEV: Radar and LiDAR BEV Fusion Learning for Anchor Box Free Object Detection System [Paper]
  • 2022 - [ST-MVDNet] Modality-Agnostic Learning for Radar-Lidar Fusion in Vehicle Detection [Oxford Radar Robotcar]CVPR [Paper]
  • 2023 - Bi-LRFusion: Bi-Directional LiDAR-Radar Fusion for 3D Dynamic Object Detection CVPR [Paper]
  • 2023 - Multi-Modal and Multi-Scale Fusion 3D Object Detection of 4D Radar and LiDAR for Autonomous Driving [Astyx] TVT [Paper]

Camera-LiDAR

Detection

  • 2021 - Joint Multi-Object Detection and Tracking with Camera-LiDAR Fusion for Autonomous Driving IROS [Paper]

Camera-LiDAR-Radar

Detection

  • 2023 - [BEV-Guide] BEV-Guided Multi-Modality Fusion for Driving Perception CVPR [Paper]
  • 2023 - DeepFusion: A Robust and Modular 3D Object Detector for Lidars, Cameras and Radars IROS [Paper]

Citation

Please use the following citation when referencing

@article{yao2023radar,
  author={Yao, Shanliang and Guan, Runwei and Huang, Xiaoyu and Li, Zhuoxiao and Sha, Xiangyu and Yue, Yong and Lim, Eng Gee and Seo, Hyungjoon and Man, Ka Lok and Zhu, Xiaohui and Yue, Yutao},
  journal={IEEE Transactions on Intelligent Vehicles}, 
  title={Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review}, 
  year={2023},
  volume={},
  number={},
  pages={1-40},
  doi={10.1109/TIV.2023.3307157}}