A Fast and Flexible Projector-Camera Calibration System

Introduction

A fast and flexible projector-camera calibration system that is single-shot-per-pose and deals with imperfect planarity of the calibration target.

Highlights:

The system explicitly deals with the noise due to imperfect planarity with a novel bundle adjustment solution.
Our calibration system applies structured light (SL) pattern keypoints rather than checkerboard corners. This strategy boosts both the number of keypoints and their spatial distribution, and hence improves calibration robustness.
Our system requires only a single shot per pose, making it practically convenient in many applications.
A practical open source projector-camera calibration software with GUI is provided to benefit the society.

For more info please refer to our journal paper and conference paper.

To reproduce journal paper results please refer to tase20 branch.
To reproduce conference paper results please refer to ismar18 branch.

Updates:

05/30/2021: Updated to MATLAB 2021a with new App Designer features. Older versions may have incompatibility issues.
06/03/2020: Journal paper accepted.
04/02/2020: Windows 64-bit executable released, no need to install MATLAB or OpenCV. See Windows 64-bit executable below.

Usage

Build from source

Install MATLAB 2021a.
Install mexOpenCV v3.3.0
Clone this repo and run calibApp.mlapp to start the App.

Windows 64-bit executable (MATLAB and OpenCV not required)

Install MATLAB Runtime 2021a for Windows 64-bit.
Download and extract ProCamCalib_v0.2.zip.
Run ProCamCalib.exe.

Tutorial

Watch a short video tutorial here.
Note our method works best for camera resolution <=640x480 and projector resolution <=1920x1080. You can resize the camera image to wdith=640 for the best quality and speed. If you want to use other resolutions please refer to this issue.

Calibration

Print a checkerboard pattern and glue it to a white board. Make sure the checkerboard pattern is at the center of the white board and its width/height is around 1/3 of the white board's width/height.
Place the white board (with checkerboard glued to it) in front of camera and projector.
Run calibApp.mlapp (or ProCamCalib.exe) to start the App.
In the App (see screenshot above), select the projector screen in the top-left Projector Control panel dropdown, then click Preview on the right of the dropdown to preview the projected structured light pattern.
Then select the camera in the top-right Camera Control panel, click Preview button to start camera review, make sure the white board is in both the camera's and the projector's FOV, i.e, the color grid covers the entire white board and the camera has a full view of the white board.
Adjust the projector color grid brightness using the Projector Brightness slider and camera parameters using the sliders in Camera Control panel. Make sure the color grid is not underexposed or overexposed.
Start to capture the images as shown in the gif below. The checkerboard and color grid images will be saved as lightGrid[i].png and colorGrid[i].png, where i is the ith position, e.g., colorGrid01.png is the color grid image at the 1st position. Although at least three poses are sufficient we highly recommend taking more, refer to Bouguet for a good example.
Once sufficient images are captured, click Calibration tab on top and select the sets you want to use for calibration, then type the printed checkerboard square size in the text box below Calibrate button. Finally click Calibrate button.

3D reconstruction

Note that we use a sparse color grid, thus the reconstructed point cloud is sparse. To obtain a dense point cloud we reconstruct both the grid nodes and edges, then interpolate it to a dense point cloud.
Click Calibration tab and load a calibration yml file by clicking Load Calibration, then select a set in the Images list and click Reconstruct.
You can choose to use Existing Mask (saved under data folder) or draw your own object mask.
Use Edges toggles color grid edge reconstruction.

Calibration comparisons

Below are results from our conference paper and can be reproduced in ismar18 branch. For the latest results, please refer to our journal paper and the results can be reproduced in tase20 branch.

Reprojection error:

Method Camera Projector Stereo

Moreno & Taubin 0.12356 1.5949 1.1311

Global homography 0.12356 5.7868 4.0928

Proposed w/o BA 0.41692 0.7105 0.5825

Proposed 0.34976 0.6352 0.5127

Method	Camera	Projector	Stereo
Moreno & Taubin	0.12356	1.5949	1.1311
Global homography	0.12356	5.7868	4.0928
Proposed w/o BA	0.41692	0.7105	0.5825
Proposed	0.34976	0.6352	0.5127

3D alignment error:

After we calibrate the camera-projector pair, we reconstruct a point cloud using 2D structured light point pairs and calibration data. To calculate reconstruction accuracy, we also capture the ground truth point cloud using an Intel RealSense F200 RGBD camera. The point cloud 3D alignment error (Euclidean distance) between the reconstructed point cloud and the ground truth point cloud are given by:

Method	Min	Max	Mean	Median	Std.
Moreno & Taubin	0.088551	55.194	8.4722	7.0756	5.9264
Global homography	0.016244	73.173	11.877	11.94	9.9919
Proposed w/o BA	0.046634	48.834	6.7798	6.8835	4.1002
Proposed	0.057853	50.807	5.5959	4.5881	4.7023

The per-point 3D alignment error can be viewed in pseudocolor:

Folder Structure

The project folder is organized as follows:

├─+Calibration                                  calibration package directory
├─+ImgProc                                      image processing package directory
├─+Reconstruct                                  3d reconstruction package directory
├─data                                          directory for data
│  ├─calibration-04-01-19_16-02-01              directory for real data, contains checkerboard/structured light images and RealSense reconstructed ply files.
│  │  ├─matlabCorners                           extracted checkerboard corners by MATLAB's detectCheckerboardPoints and warped corners by the global homography             
│  │  └─results                                 calibration results generated by the four methods mention in the paper with real data
│  └─simulation                                 directory for simulation (synthetic) data
│      └─results                                calibration results generated by the four methods mention in the paper with synthetic data
├─doc                                           directory for documentation
└─README.md                                     this file

Citation

Please cite these papers in your publications if it helps your research:

@article{huang2020flexible,
    title={A Fast and Flexible Projector-Camera Calibration System}, 
    author={Huang, Bingyao and Tang, Ying and Ozdemir, Samed and Ling, Haibin},
    journal={IEEE Transactions on Automation Science and Engineering}, 
    year={2020},
    doi = {10.1109/TASE.2020.2994223}
}

@inproceedings{huang2018single,
    title={A Single-shot-per-pose Camera-Projector Calibration System For Imperfect Planar Targets},
    author={Huang, Bingyao and Ozdemir, Samed and Tang, Ying and Liao, Chunyuan and Ling, Haibin},
    booktitle={2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
    pages={15--20},
    year={2018},
    organization={IEEE}
}

@inproceedings{huang2014fast,
    title={Fast 3D reconstruction using one-shot spatial structured light},
    author={Huang, Bingyao and Tang, Ying},
    booktitle={2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC)},
    pages={531--536},
    year={2014},
    organization={IEEE}
}

License

This software is freely available for non-profit non-commercial use, and may be redistributed under the conditions in license.

BingyaoHuang/single-shot-pro-cam-calib