/Lightweight-Deformable-GS

[Just 4 Fun] Deformable-GS with less storage, 150+FPS, and sota quality.

Primary LanguagePythonMIT LicenseMIT

Lightweight Deformable 3D Gaussian

Project page | Paper | Vanilla Code

Teaser image

This repository serves as an enhancement project for the paper 'Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction'. By implementing a densification trick, we can achieve a halved number of Gaussians and higher FPS without compromising rendering quality.

Dataset

In our paper, we use:

  • synthetic dataset from D-NeRF.
  • real-world dataset from NeRF-DS and Hyper-NeRF.
  • The dataset in the supplementary materials comes from DeVRF.

We organize the datasets as follows:

├── data
│   | D-NeRF 
│     ├── hook
│     ├── standup 
│     ├── ...
│   | NeRF-DS
│     ├── as
│     ├── basin
│     ├── ...
│   | HyperNeRF
│     ├── interp
│     ├── misc
│     ├── vrig

I have identified an inconsistency in the D-NeRF's Lego dataset. Specifically, the scenes corresponding to the training set differ from those in the test set. This discrepancy can be verified by observing the angle of the flipped Lego shovel. To meaningfully evaluate the performance of our method on this dataset, I recommend using the validation set of the Lego dataset as the test set. See more in D-NeRF dataset used in Deformable-GS

Run

Environment

git clone https://github.com/ingra14m/Lightweight-Deformable-GS --recursive
cd Lightweight-Deformable-GS

conda create -n deformable_gaussian_env python=3.8
conda activate deformable_gaussian_env

# install pytorch
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116

# install dependencies
pip install -r requirements.txt

Train

D-NeRF:

python train.py -s path/to/your/d-nerf/dataset -m output/exp-name --eval --is_blender

NeRF-DS/HyperNeRF:

python train.py -s path/to/your/real-world/dataset -m output/exp-name --eval --iterations 20000

Render & Evaluation

python render.py -m output/exp-name --mode render
python metrics.py -m output/exp-name

We provide several modes for rendering:

  • render: render all the test images
  • time: time interpolation tasks for D-NeRF dataset
  • all: time and view synthesis tasks for D-NeRF dataset
  • view: view synthesis tasks for D-NeRF dataset
  • original: time and view synthesis tasks for real-world dataset

Results

FPS is tested on 3090.

D-NeRF

$\tau_g=0.0007$

PSNR SSIM LPIPS(VGG) FPS Mem(MB) Num.
bouncing 41.79 0.9959 0.0082 111 15.95 67417
hell 41.36 0.9869 0.0253 288 4.19 17709
hook 36.82 0.9852 0.0168 128 14.50 61321
jump 37.56 0.9895 0.0135 223 7.36 31118
mutant 42.10 0.9943 0.0066 109 17.04 72028
standup 43.85 0.9941 0.0085 197 6.84 28895
trex 37.58 0.9927 0.0106 74 22.90 96816
average 40.15 0.9912 0.0128 162 12.68 53614
paper 40.43 0.9918 0.0116 70 26.71 131428

$\tau_g=0.0006$

PSNR SSIM LPIPS(VGG) FPS Mem(MB) Num.
bouncing 41.16 0.9955 0.0088 100 18.46 78033
hell 41.39 0.9866 0.0253 241 5.27 22288
hook 36.93 0.9853 0.0165 105 17.94 75831
jump 37.76 0.9896 0.0135 147 9.33 39457
mutant 42.27 0.9946 0.0060 93 21.55 91125
standup 44.17 0.9944 0.0079 163 8.74 36939
trex 38.03 0.9931 0.0099 59 27.56 116521
average 40.25 0.9913 0.0125 135 15.55 65742

$\tau_g=0.0005$

PSNR SSIM LPIPS(VGG) FPS Mem(MB) Num.
bouncing 40.85 0.9952 0.0091 80 23.90 101031
hell 41.39 0.9866 0.0247 204 6.65 27704
hook 37.17 0.9859 0.0158 68 23.15 97876
jump 37.68 0.9894 0.0135 128 11.87 49986
mutant 43.32 0.9948 0.0055 64 27.5 116276
standup 44.18 0.9946 0.0076 129 11.15 47141
trex 37.76 0.9929 0.0099 47 34.7 146707
average 40.19 0.9913 0.0123 103 19.84 83817

NeRF-DS

PSNR SSIM LPIPS(VGG) FPS Mem(MB) Num.
as 26.42 0.8865 0.1804 86 24.34 102918
basin 19.72 0.7948 0.1881 63 33.81 142943
bell 25.49 0.8466 0.1580 53 43.29 183499
cup 24.78 0.8895 0.1540 58 27.97 118261
plate 20.54 0.8143 0.2184 71 29.26 123712
press 25.67 0.8637 0.1931 76 27.63 116800
sieve 25.37 0.8704 0.1489 71 28.45 120280
average 24.00 0.8523 0.1773 68 30.68 129773
paper 24.11 0.8524 0.1769 35 56.78 221428

BibTex

@article{yang2023deformable3dgs,
    title={Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction},
    author={Yang, Ziyi and Gao, Xinyu and Zhou, Wen and Jiao, Shaohui and Zhang, Yuqing and Jin, Xiaogang},
    journal={arXiv preprint arXiv:2309.13101},
    year={2023}
}

And thanks to the authors of 3D Gaussians for their excellent code, please consider also cite this repository:

@Article{kerbl3Dgaussians,
      author       = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
      title        = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
      journal      = {ACM Transactions on Graphics},
      number       = {4},
      volume       = {42},
      month        = {July},
      year         = {2023},
      url          = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}
}