/Awesome-Deep-Camera-Calibration

Deep Learning for Camera Calibration and Beyond: A Survey

Awesome-Deep-Camera-Calibration

arXiv Survey Maintenance PR's Welcome GitHub license

Overview

Popular calibration objectives, models, and extended applications in camera calibration

Our repository provides：

1) a time line of the development in the related research community 2) a taxonomy of the survey 3) a public and holistic benchmark 4) collects deep learning-based camera calibration Publication, Objective, Network 5) dataset summary

More content and details can be found in our Survey Paper: Deep Learning for Camera Calibration and Beyond: A Survey.

📣News

  1. The survey of arXiv version is online.

🚩Contents

  1. Basics
  2. Taxonomy
  3. Benchmark
  4. Methods
  5. Dataset
  6. Citation
  7. 知乎导读

🚩Basics

🚩Taxonomy

Overview The structural and hierarchical taxonomy of camera calibration with deep learning. Some classical methods are listed under each category.

Overview A concise milestone of deep learning-based camera calibration methods. We classify all methods based on the uncalibrated camera model and its extended applications: standard model, distortion model, cross-view model, and cross-sensor model.

🚩Benchmark

Overview Overview of our collected benchmark, which covers all models reviewed in this survey. In this dataset, the image and video derive from diverse cameras under different environments. The accurate ground truth and label are provided for each data. We will make this benchmark available soon at Google Drive.

🚩Methods

Year Publication Title Abbreviation Objective Platform Network
2015 ICIP Deepfocal: A method for direct focal length estimation DeepFocal Intrinsics Caffe AlexNet
2015 ICCV Posenet: A convolutional network for real-time 6-dof camera relocalization PoseNet Extrinsics Caffe GoogLeNet
2016 BMVC Horizon lines in the wild DeepHorizon Extrinsics Caffe GoogLeNet
2016 CVPR Detecting vanishing points using global image context in a non-manhattan world DeepVP Extrinsics Caffe AlexNet
2016 ACCV Radial lens distortion correction using convolutional neural networks trained with synthesized images Rong et al. Distortion coefficients Caffe AlexNet
2016 RSSW Deep image homography estimation DHN Projection matrixs Caffe VGG
2017 CVPR Clkn: Cascaded lucas-kanade networks for image alignment CLKN Projection matrixs Torch CNN + Lucas-Kanade layer
2017 ICCVW Homography estimation from image pairs with hierarchical convolutional networks HierarchicalNet Projection matrixs TensorFlow VGG
2017 CVPR Unrolling the Shutter: CNN to Correct Motion Distortions URS-CNN Undistortion Torch CNNs
2017 IV RegNet: Multimodal sensor registration using deep neural networks RegNet Camera + LiDAR Caffe CNNs
2018 CVPR A perceptual measure for deep single image camera calibration Hold-Geoffroy et al. Intrinsics + Extrinsics DenseNet
2018 CVMP DeepCalib: a deep learning approach for automatic intrinsic calibration of wide field-of-view cameras DeepCalib Intrinsics + Distortion coefficients TensorFlow Inception-V3
2018 ECCV Fisheyerecnet: A multi-context collaborative deep network for fisheye image rectification FishEyeRecNet Distortion coefficients Caffe VGG
2018 ICPR Radial lens distortion correction by adding a weight layer with inverted foveal models to convolutional neural networks Shi et al. Distortion coefficients PyTorch ResNet
2018 ECCV Deep fundamental matrix estimation DeepFM Projection matrixs PyTorch ResNet
2018 ECCVW Deep fundamental matrix estimation without correspondences Poursaeed et al. Projection matrixs CNNs
2018 RAL Unsupervised deep homography: A fast and robust homography estimation model UDHN Projection matrixs TensorFlow VGG
2018 ACCV Rethinking planar homography estimation using perspective fields PFNet Projection matrixs TensorFlow FCN
2018 IROS CalibNet: Geometrically Supervised Extrinsic Calibration using 3D Spatial Transformer Networks CalibNet Camera + LiDAR TensorFlow ResNet
2018 ICRA DeepVP: Deep Learning for Vanishing Point Detection on 1 Million Street View Images Chang et al. Standard Matconvnet AlexNet
2019 CVPR Deep single image camera calibration with radial distortion Lopez et al. Intrinsics + Extrinsics + Distortion coefficients PyTorch DenseNet
2019 ICCV UprightNet: geometry-aware camera orientation estimation from single images UprightNet Extrinsics PyTorch U-Net
2019 IROS Degeneracy in self-calibration revisited and a deep learning solution for uncalibrated slam Zhuang et al. Intrinsics + Distortion coefficients PyTorch ResNet
2019 PRL Self-Supervised deep homography estimation with invertibility constraints SSR-Net Projection matrixs PyTorch ResNet
2019 ICCVW A geometric approach to obtain a bird's eye view from an image Abbas et al. Projection matrixs TensorFlow CNNs
2019 TCSVT DR-GAN: Automatic radial distortion rectification using conditional GAN in real-time DR-GAN Undistortion TensorFlow GANs
2019 TCSVT Distortion rectification from static to dynamic: A distortion sequence construction perspective STD Undistortion TensorFlow GANs
2019 VR Deep360Up: A deep learning-based approach for automatic VR image upright adjustment Deep360Up Extrinsics DenseNet
2019 JVCIR Unsupervised fisheye image correction through bidirectional loss with geometric prior UnFishCor Distortion coefficients TensorFlow VGG
2019 CVPR Blind geometric distortion correction on images through deep learning BlindCor Undistortion PyTorch U-Net
2019 CVPR Learning structure-and-motion-aware rolling shutter correction RSC-Net Undistortion PyTorch ResNet
2019 CVPR Learning to calibrate straight lines for fisheye image rectification Xue et al. Distortion coefficients PyTorch ResNet
2019 ICCV Learning perspective undistortion of portraits Zhao et al. Intrinsics + Undistortion VGG + U-Net
2019 NeurIPS NeurVPS: Neural Vanishing Point Scanning via Conic Convolution NeurVPS Standard PyTorch CNNs
2020 CVPR End-to-end camera calibration for broadcast videos Sha et al. Projection matrixs TensorFlow Siamese-Net + U-Net
2020 ECCV Neural geometric parser for single image camera calibration Lee et al. Intrinsics + Extrinsics PointNet + CNNs
2020 ICRA Learning camera miscalibration detection MisCaliDet Average pixel position difference TensorFlow CNNs
2020 WACV DeepPTZ: deep self-calibration for PTZ cameras DeepPTZ Intrinsics + Extrinsics + Distortion coefficients PyTorch Inception-V3
2020 CVPR Deep homography estimation for dynamic scenes MHN Projection matrixs TensorFlow VGG
2020 ACMMM SRHEN: Stepwise-Refining Homography Estimation Network via Parsing Geometric Correspondences in Deep Latent Space SRHEN Projection matrixs CNNs
2020 ECCV 360∘ camera alignment via segmentation Davidson et al. Extrinsics FCN
2020 ECCV Content-aware unsupervised deep homography estimation CA-UDHN Projection matrixs PyTorch FCN + ResNet
2020 IROS Deep keypoint-based camera pose estimation with geometric constraints DeepFEPE Extrinsics PyTorch VGG + PointNet
2020 TIP Model-free distortion rectification framework bridged by distortion distribution map DDM Undistortion Tensorflow GANs
2020 TIP Deep face rectification for 360° dual-fisheye cameras Li et al. Undistortion CNNs
2020 ICPR Position-aware and symmetry enhanced GAN for radial distortion correction PSE-GAN Undistortion GANs
2020 ICIP A simple yet effective pipeline for radial distortion correction RDC-Net Undistortion PyTorch ResNet
2020 ICASSP Self-supervised deep learning for fisheye image rectification FE-GAN Undistortion PyTorch GANs
2020 CVPR RDCFace: radial distortion correction for face recognition RDCFace Undistortion ResNet
2020 arXiv Fisheye distortion rectification from deep straight lines LaRecNet Distortion coefficients PyTorch ResNet
2020 CVPR Height and uprightness invariance for 3d prediction from a single view Baradad et al. Intrinsics + Extrinsics PyTorch CNNs
2020 CVPR What does plate glass reveal about camera calibration? Zheng et al. Intrinsics + Extrinsics CNNs
2020 ECCV Single view metrology in the wild Zhu et al. Intrinsics + Extrinsics PyTorch CNNs + PointNet
2020 CVPR Deep Shutter Unrolling Network DeepUnrollNet Undistortion PyTorch FCN
2020 RAL RGGNet: Tolerance Aware LiDAR-Camera Online Calibration With Geometric Deep Learning and Generative Model RGGNet Camera + LiDAR Tensorflow ResNet
2020 IROS CalibRCNN: Calibrating Camera and LiDAR by Recurrent Convolutional Neural Network and Geometric Constraints CalibRCNN Camera + LiDAR Tensorflow RNN
2020 ICRA Online Camera-LiDAR Calibration with Sensor Semantic Information SSI-Calib Camera-LiDAR Tensorflow CNNs
2020 arXiv SOIC: Semantic Online Initialization and Calibration for LiDAR and Camera SOIC Camera-LiDAR - ResNet+PointRCNN
2020 ICPR NetCalib: A Novel Approach for LiDAR-Camera Auto-calibration Based on Deep Learning NetCalib Camera-LiDAR PyTorch CNNs
2021 TCI Online training of stereo self-calibration using monocular depth estimation StereoCaliNet Extrinsics PyTorch U-Net
2021 ICCV CTRL-C: Camera calibration TRansformer with Line-Classification CTRL-C Intrinsics + Extrinsics PyTorch Transformer
2021 ICCVW Deep single fisheye image camera calibration for over 180-degree projection of field of view Wakai et al. Intrinsics + Extrinsics DenseNet
2021 TIP A deep ordinal distortion estimation approach for distortion rectification OrdianlDistortion Distortion coefficients TensorFlow CNNs
2021 TCSVT Revisiting radial distortion rectification in polar-coordinates: A new and efficient learning perspective PolarRecNet Undistortion PyTorch VGG + U-Net
2021 PRL DQN-based gradual fisheye image rectification DQN-RecNet Undistortion PyTorch VGG
2021 CVPR Practical wide-angle portraits correction with deep structured models Tan et al. Undistortion PyTorch U-Net
2021 CVPR Progressively complementary network for fisheye image rectification using appearance flow PCN Undistortion PyTorch U-Net
2021 ICCV Multi-level curriculum for training a distortion-aware barrel distortion rectification model DaRecNet Undistortion TensorFlow U-Net
2021 CVPR Deep Lucas-Kanade homography for multimodal image alignment DLKFM Projection matrixs TensorFlow Siamese-Net
2021 ICCV LocalTrans: A multiscale local transformer network for cross-resolution homography estimation LocalTrans Projection matrixs PyTorch Transformer
2021 ICCV Motion basis learning for unsupervised deep homography estimation with subspace projection BasesHomo Projection matrixs PyTorch ResNet
2021 ICIP Fast and accurate homography estimation using extendable compression network ShuffleHomoNet Projection matrixs TensorFlow ShuffleNet
2021 TCSVT Depth-aware multi-grid deep homography estimation with contextual correlation DAMG-Homo Projection matrixs TensorFlow CNNs
2021 BMVC A simple approach to image tilt correction with self-attention MobileNet for smartphones SA-MobileNet Extrinsics TensorFlow MobileNet
2021 ICCV SPEC: Seeing people in the wild with an estimated camera SPEC Intrinsics + Extrinsics PyTorch ResNet
2021 CVPR Wide-Baseline Relative Camera Pose Estimation with Directional Learning DirectionNet Extrinsics TensorFlow U-Net
2021 CVPR Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes JCD Undistortion PyTorch FCN
2021 CVPRW LCCNet: LiDAR and Camera Self-Calibration using Cost Volume Network LCCNet Camera + LiDAR PyTorch CNNs
2021 Sensors CFNet: LiDAR-Camera Registration Using Calibration Flow Network CFNet Camera + LiDAR PyTorch FCN
2021 ICCV Inverting a Rolling Shutter Camera: Bring Rolling Shutter Images to High Framerate Global Shutter Video Fan et al. Distortion PyTorch U-Net
2021 ICCV SUNet: Symmetric Undistortion Network for Rolling Shutter Correction SUNet Distortion PyTorch DenseNet+ResNet
2021 IROS SemAlign: Annotation-Free Camera-LiDAR Calibration with Semantic Alignment Loss SemAlign Camera-LiDAR PyTorch CNNs
2022 CVPR Deep vanishing point detection: Geometric priors make dataset variations vanish DVPD Extrinsics PyTorch CNNs
2022 ICRA Self-supervised camera self-calibration from video Fang et al. Intrinsics + Extrinsics PyTorch CNNs
2022 ICASSP Camera calibration through camera projection loss CPL Intrinsics + Extrinsics TensorFlow Inception-V3
2022 CVPR Iterative Deep Homography Estimation IHN Projection matrixs PyTorch Siamese-Net
2022 CVPR Unsupervised Homography Estimation with Coplanarity-Aware GAN HomoGAN Projection matrixs PyTorch GANs
2022 CVPR Semi-Supervised Wide-Angle Portraits Correction by Multi-Scale Transformer SS-WPC Undistortion PyTorch Transformer
2022 CVPR Learning Adaptive Warping for Real-World Rolling Shutter Correction AW-RSC Undistortion CNNs
2022 CVPR EvUnroll: Neuromorphic Events based Rolling Shutter Image Correction EvUnroll Undistortion PyTorch U-Net
2022 CVPR Learning to Detect Scene Landmarks for Camera Localization Do et al. Extrinsics PyTorch ResNet
2022 CVPR DiffPoseNet: Direct Differentiable Camera Pose Estimation DiffPoseNet Extrinsics PyTorch CNNs + LSTM
2022 CVPR SceneSqueezer: Learning to Compress Scene for Camera Relocalization SceneSqueezer Extrinsics PyTorch Transformer
2022 arXiv FishFormer: Annulus Slicing-based Transformer for Fisheye Rectification with Efficacy Domain Exploration FishFormer Undistortion PyTorch Transformer
2022 CVPR Focal Length and Object Pose Estimation via Render and Compare FocalPose Intrinsics + Extrinsics PyTorch CNNs
2022 arXiv DXQ-Net: Differentiable LiDAR-Camera Extrinsic Calibration Using Quality-aware Flow DXQ-Net Camera + LiDAR PyTorch CNNs + RNNs
2022 ITSC SST-Calib: Simultaneous Spatial-Temporal Parameter Calibration between LIDAR and Camera SST-Calib Camera + LiDAR PyTorch CNNs
2022 IROS Learning-Based Framework for Camera Calibration with Distortion Correction and High Precision Feature Detection CCS-Net Undistortion PyTorch UNet
2022 TIP SIR: Self-supervised image rectification via seeing the same scene from multiple different lenses SIR Undistortion PyTorch ResNet
2022 TIV ATOP: An Attention-to-Optimization Approach for Automatic LiDAR-Camera Calibration Via Cross-Modal Object Matching ATOP Camera + LiDAR CNNs
2022 ICRA FusionNet: Coarse-to-Fine Extrinsic Calibration Network of LiDAR and Camera with Hierarchical Point-pixel Fusion FusionNet Camera + LiDAR PyTorch CNNs+PointNet
2022 TIM Keypoint-Based LiDAR-Camera Online Calibration With Robust Geometric Network RKGCNet Camera + LiDAR PyTorch CNNs+PointNet
2022 ECCV Rethinking generic camera models for deep single image camera calibration to recover rotation and fisheye distortion GenCaliNet Intrinsics + Extrinsics + Distortion coefficients DenseNet
2022 PAMI Content-Aware Unsupervised Deep Homography Estimation and Beyond Liu et al. Projection matrixs PyTorch ResNet

🚩Datasets

Name Publication Real/Synthetic Image/Video Objectives Dataset
KITTI CVPR Real Video Base Dataset
MS-COCO ECCV Real Image Base Dataset
SUN360 CVPR Real Image Base Dataset
Places2 PAMI Real Image Base Dataset
CelebA ICCV Real Image Base Dataset
1DSfM ECCV Real Image Focal Length Dataset
Cambridge Landmarks ICCV Real Video Extrinsics Dataset
HLW BMVC Real Image Horizon Line Dataset
YUD ECCV Real Image Vanishing Point Dataset
ECD ECCV Real Image Vanishing Point Dataset
SU3 Wireframe ICCV Synthetic Image Vanishing Point Dataset
ScanNet CVPR Real Video Extrinsics Dataset
Indoor-6 CVPR Real Image Extrinsics Dataset
DeepVP ICRA Real Image Vanishing Point Dataset
CAHomo ECCV Real Video Homography Dataset
MHN CVPR Real Video Homography Dataset
UDIS TIP Real Video Homography Dataset
Carla-RS CVPR Synthetic Video RS-Distortion Dataset
Fastec-RS CVPR Synthetic Video RS-Distortion Dataset
BS-RSC CVPR Real Video RS-Distortion Dataset
GEV-RS CVPR Real Video RS-Distortion Dataset
LMS ICASSP Both Video Radial Distortion Dataset
SS-WPC CVPR Real Image Radial Distortion Dataset

📜License

The survey and benchmark are only made available for academic research purpose.

📚Citation

@article{kang2023deep,
Author = {Kang Liao and Lang Nie and Shujuan Huang and Chunyu Lin and Jing Zhang and Yao Zhao and Moncef Gabbouj and Dacheng Tao},
Title = {Deep Learning for Camera Calibration and Beyond: A Survey},
Year = {2023},
Journal = {arXiv:2303.10559}
}

🚩知乎导读

导读链接

📭Contact

kang_liao@bjtu.edu.cn