/awesome-state-of-depth-completion

Current state of supervised and unsupervised depth completion methods

Awesome Depth Completion

Table of Contents

About Sparse-to-Dense Depth Completion

In the sparse-to-dense depth completion problem, one wants to infer the dense depth map of a 3-D scene given an RGB image and its corresponding sparse reconstruction in the form of a sparse depth map obtained either from computational methods such as SfM (Strcuture-from-Motion) or active sensors such as lidar or structured light sensors.

Example 1: VOID dataset (indoor VIO)

Input RGB image Sparse point cloud Output point cloud from KBNet

Example 2: KITTI dataset (outdoor lidar)

Input RGB image Output point cloud from ScaffNet

Current State of Depth Completion Methods

Here we compile both unsupervised/self-supervised (monocular and stereo) and supervised methods published in recent conferences and journals on the VOID (Wong et. al., 2020) and KITTI (Uhrig et. al., 2017) depth completion benchmarks. Our ranking considers all four metrics rather than just RMSE.

Quick Links

Unsupervised VOID Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Monitored Distillation for Positive Congruent Depth Completion ECCV 2022 PyTorch 36.42 87.78 19.18 43.83
Unsupervised Depth Completion with Calibrated Backprojection Layers ICCV 2021 PyTorch 39.80 95.86 21.16 49.72
Learning Topology from Synthetic Data for Unsupervised Depth Completion RA-L & ICRA 2021 Tensorflow/PyTorch 60.68 122.01 35.24 67.34
Unsupervised Depth Completion from Visual Inertial Odometry RA-L & ICRA 2020 Tensorflow/PyTorch 85.05 169.79 48.92 104.02
Dense depth posterior (ddp) from single image and sparse range CVPR 2019 Tensorflow 151.86 222.36 74.59 112.36
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular Camera ICRA 2019 PyTorch 178.85 243.84 80.12 107.69

Supervised VOID Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
CostDCNet: Cost Volume based Depth Completion for a Single RGB-D Image ECCV 2022 PyTorch 25.84 76.28 12.19 32.13
Non-Local Spatial Propagation Network for Depth Completion ECCV 2020 PyTorch 26.74 79.12 12.70 33.88
Monitored Distillation for Positive Congruent Depth Completion ECCV 2022 PyTorch 29.67 79.78 14.84 37.88
PENet: Towards Precise and Efficient Image Guided Depth Completion (PENet) ICRA 2021 PyTorch 34.61 82.01 18.89 40.36
A Multi-Scale Guided Cascade Hourglass Network for Depth Completion WACV 2020 PyTorch 43.57 109.94 23.44 52.09
PENet: Towards Precise and Efficient Image Guided Depth Completion (ENet) ICRA 2021 PyTorch 46.90 94.35 26.78 52.58
Scanline Resolution-Invariant Depth Completion Using a Single Image and Sparse LiDAR Point Cloud RA-L & IROS 2021 N/A 59.40 181.42 19.37 46.56

Unsupervised KITTI Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Unsupervised Depth Completion with Calibrated Backprojection Layers ICCV 2021 PyTorch 256.76 1069.47 1.02 2.95
Learning Topology from Synthetic Data for Unsupervised Depth Completion RA-L & ICRA 2021 Tensorflow 280.76 1121.93 1.15 3.30
Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data ACCV 2020 PyTorch 280.42 1095.26 1.19 3.53
Unsupervised Depth Completion from Visual Inertial Odometry RA-L & ICRA 2020 Tensorflow 299.41 1169.97 1.20 3.56
A Surface Geometry Model for LiDAR Depth Completion RA-L & ICRA 2021 Tensorflow 298.3 1239.84 1.21 3.76
Dense depth posterior (ddp) from single image and sparse range CVPR 2019 Tensorflow 343.46 1263.19 1.32 3.58
DFuseNet: Deep Fusion of RGB and Sparse Depth Information for Image Guided Dense Depth Completion ITSC 2019 PyTorch 429.93 1206.66 1.79 3.62
In Defense of Classical Image Processing: Fast Depth Completion on the CPU CRV 2018 Python 302.60 1288.46 1.29 3.78
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular Camera ICRA 2019 PyTorch 350.32 1299.85 1.57 4.07
Semantically Guided Depth Upsampling GCPR 2016 N/A 605.47 2312.57 2.05 7.38

Supervised KITTI Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Dynamic Spatial Propagation Network for Depth Completion AAAI 2022 N/A 192.71 709.12 0.82 1.88
Non-Local Spatial Propagation Network for Depth Completion ECCV 2020 PyTorch 199.5 741.68 0.84 1.99
CompletionFormer: Depth Completion with Convolutions and Vision Transformers CVPR 2023 PyTorch 203.45 708.87 0.88 2.01
SemAttNet: Towards Attention-based Semantic Aware Guided Depth Completion IEEE Access 2022 PyTorch 205.49 709.41 0.90 2.03
RigNet: Repetitive Image Guided Network for Depth Completion ECCV 2022 N/A 203.25 712.66 0.90 2.08
MFF-Net: Towards Efficient Monocular Depth Completion with Multi-modal Feature Fusion RAL 2023 N/A 208.11 719.85 0.94 2.21
CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion AAAI 2020 N/A 209.28 743.69 0.90 2.07
Dense depth posterior (ddp) from single image and sparse range CVPR 2019 Tensorflow 203.96 832.94 0.85 2.10
Adaptive context-aware multi-modal network for depth completion TIP 2021 PyTorch 206.80 732.99 0.90 2.08
PENet: Towards Precise and Efficient Image Guided Depth Completion ICRA 2021 PyTorch 210.55 730.08 0.94 2.17
Monitored Distillation for Positive Congruent Depth Completion ECCV 2022 PyTorch 218.60 785.06 0.92 2.11
MDANet: Multi-Modal Deep Aggregation Network for Depth Completion ICRA 2021 PyTorch 214.99 738.23 0.99 2.12
A Cascade Dense Connection Fusion Network for Depth Completion BMVC 2022 N/A 216.05 738.26 0.99 2.18
FCFR-Net: Feature Fusion based Coarse- to-Fine Residual Learning for Depth Completion AAAI 2021 N/A 217.15 735.81 0.98 2.20
Learning Guided Convolutional Network for Depth Completion TIP 2020 PyTorch 218.83 736.24 0.99 2.25
DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network ICRA 2021 N/A 214.13 755.41 0.96 2.25
A Multi-Scale Guided Cascade Hourglass Network for Depth Completion WACV 2020 PyTorch 220.41 762.19 0.98 2.30
Sparse and noisy LiDAR completion with RGB guidance and uncertainty MVA 2019 PyTorch 215.02 772.87 0.93 2.19
A Multi-Scale Guided Cascade Hourglass Network for Depth Completion WACV 2020 N/A 220.41 762.19 0.98 2.30
Learning Joint 2D-3D Representations for Depth Completion ICCV 2019 N/A 221.19 752.88 1.14 2.34
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image CVPR 2019 PyTorch 226.50 758.38 1.15 2.56
Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints ICCV 2019 N/A 235.17 777.05 1.13 2.42
Scanline Resolution-Invariant Depth Completion Using a Single Image and Sparse LiDAR Point Cloud RA-L & IROS 2021 N/A 233.34 809.09 1.06 2.57
Confidence propagation through cnns for guided sparse depth regression PAMI 2019 PyTorch 233.26 829.98 1.03 2.60
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular Camera ICRA 2019 PyTorch 249.95 814.73 1.21 2.80
Uncertainty-Aware CNNs for Depth Completion: Uncertainty from Beginning to End CVPR 2020 PyTorch 251.77 960.05 1.05 3.37
Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation 3DV 2019 N/A 234.81 917.64 0.95 2.17
Depth coefficients for depth completion CVPR 2019 N/A 252.21 988.38 1.13 2.87
Depth estimation via affinity learned with convolutional spatial propagation network ECCV 2018 N/A 279.46 1019.64 1.15 2.93
Learning morphological operators for depth completion ACIVS 2019 N/A 310.49 1045.45 1.57 3.84
Sparsity Invariant CNNs 3DV 2017 Tensorflow 416.14 1419.75 1.29 3.25
Deep Convolutional Compressed Sensing for LiDAR Depth Completion ACCV 2018 Tensorflow 439.48 1325.37 3.19 59.39