On the Error Analysis of 3D Gaussian Splatting and an Optimal Projection Strategy

ECCV 2024

Letian Huang Jiayang Bai Jie Guo^* Yuanqi Li Yanwen Guo
Nanjing University

News

[2024.07.16] 🎈 We release the code.

[2024.07.05] Birthday of the repository.

TODO List

Release the code (submodule for the pinhole camera's rasterization).
Release the submodule for the panorama's rasterization.
Release the submodule for the fisheye camera's rasterization.
Code optimization (as mentioned in the limitation of the paper, the current CUDA implementation is slow and needs optimization).

Motivation

We derive the mathematical expectation of the projection error (Top left), visualize the graph of the error function under two distinct domains and analyze when this function takes extrema through methods of function optimization (Top right). We further derive the projection error function with respect to image coordinates and focal length through the coordinate transformation between image coordinates and polar coordinates and visualize this function, with the left-to-right sequence corresponding to the 3D-GS rendered images under long focal length, 3D-GS rendered images under short focal length, the error function under long focal length, and the error function under short focal length (Below).

Pipeline

Illustration of the rendering pipeline for our Optimal Gaussian Splatting and the projection of 3D Gaussian Splatting. The blue box depicts the projection process of the original 3D-GS, which straightforwardly projects all Gaussians onto the same projection plane. In contrast, the red box illustrates our approach, where we project individual Gaussians onto corresponding tangent planes.

Installation

Clone the repository and create an anaconda environment using

git clone git@github.com:LetianHuang/op43dgs.git
cd op43dgs

SET DISTUTILS_USE_SDK=1 # Windows only
conda env create --file environment.yml
conda activate op43dgs

The repository contains several submodules, thus please check it out with

# Pinhole
pip install submodules/diff-gaussian-rasterization-pinhole

# Panorama
pip install submodules/diff-gaussian-rasterization-panorama

# Fisheye
pip install submodules/diff-gaussian-rasterization-fisheye

Dataset

Mip-NeRF 360 Dataset

Please download the data from the Mip-NeRF 360.

Tanks & Temples dataset

Please download the data from the 3D Gaussian Splatting.

Deep Blending

Please download the data from the 3D Gaussian Splatting.

Training and Evaluation

By default, the trained models use all available images in the dataset. To train them while withholding a test set for evaluation, use the --eval flag. This way, you can render training/test sets and produce error metrics as follows:

python train.py -s <path to COLMAP or NeRF Synthetic dataset> --eval # Train with train/test split
python render.py -m <path to trained model> --fov_ratio 1 # Generate renderings

Command Line Arguments for render.py

--model_path / -m

Path to the trained model directory you want to create renderings for.

--skip_train

Flag to skip rendering the training set.

--skip_test

Flag to skip rendering the test set.

--quiet

Flag to omit any text written to standard out pipe.

--fov_ratio

Focal length reduction ratios.

Acknowledgements

This project is built upon 3DGS. Please follow the license of 3DGS. We thank all the authors for their great work and repos.

Citation

If you find this work useful in your research, please cite:

@article{huang2024erroranalysis3dgaussian,
    title={On the Error Analysis of 3D Gaussian Splatting and an Optimal Projection Strategy},
    author={Letian Huang and Jiayang Bai and Jie Guo and Yuanqi Li and Yanwen Guo},
    journal={arXiv preprint arXiv:2402.00752},
    year={2024}
}