/Fast-Learning-NeRF

[TIP'2023] Fast Learning Radiance Fields by Shooting Much Fewer Rays, a general strategy to speed up the learning of radiance field

Primary LanguagePython

Fast Learning Radiance Fields by Shooting Much Fewer Rays

Wenyuan Zhang1, Ruofan Xing1, Yunfan Zeng1, Yu-Shen Liu1, Kanle Shi2, Zhizhong Han3

1Tsinghua University, 2Kuaishou Technology, 3Wayne State University

In this work, we introduce a general strategy to speed up the learning procedure for almost all radiance fields based methods by shooting much fewer rays.

Setup

Python 3 dependencies:

  • pytorch
  • matplotlib
  • numpy
  • imageio
  • configargparse

Specifically, our experiments are conducted on six radiance field based methods, including NeRF, NeRF++, Plenoxels, Mip-NeRF 360, Instant-NGP and NeuS. We finetuned their official codes and apply our method to the baseline methods. To setup each individual environment for each experiment, please refer to the project pages of the baseline methods.

Data preparation

We use five datasets named synthetic dataset, llff dataset, lf dataset, tanks_and_temples dataset, which are widely used in radiance fields based methods. You can download the four datasets from the following links.


Running code

Running NeRF-based Method:

cd nerf-ours

Training

CUDA_VISIBLE_DEVICES=0 python run_nerf.py --config configs/lego.txt

Testing

CUDA_VISIBLE_DEVICES=0 python run_nerf.py --config configs/lego.txt --ft_path logs/paper_lego/011.tar --render_only --render_test

Running NeRF++-based Method:

cd nerf++-ours

Training

CUDA_VISIBLE_DEVICES=0 python ddp_train_nerf.py --config configs/tanks_and_temples/tat_training_truck.txt

Testing

CUDA_VISIBLE_DEVICES=0 python ddp_test_nerf.py --config configs/tanks_and_temples/tat_training_truck.txt --render_splits test

Running Plenoxels-based Method:

cd plenoxels-ours/opt

Training

CUDA_VISIBLE_DEVICES=0 python opt.py data/nerf_synthetic/lego -t ckpt/paper_lego -c configs/syn.json

Testing

CUDA_VISIBLE_DEVICES=0 python render_imgs.py ckpt/paper_lego/ckpt.npz data/nerf_synthetic/lego

Running Instant-NGP-based Method:

cd ngp-ours

Training & Testing

CUDA_VISIBLE_DEVICES=0 python python main_nerf.py data/TanksAndTemple/Family --workspace log/tnt/Family -O --bound 1.0 --scale 0.33 --dt_gamma 0

Some Visualization Results

Visulization of Quadtree Subdivision Procedure

Visualization of Comparison on Synthetic Dataset

Visualization of Comparison on LF Dataset

Visualization of Comparison on Tanks And Temples Dataset

Visualization of Comparison on Mip-NeRF 360 Dataset

Visualization of Comparison on DTU Dataset

Citation

@article{zhang2023fast,
      title={Fast Learning Radiance Fields by Shooting Much Fewer Rays},
      author={Zhang, Wenyuan and Xing, Ruofan and Zeng, Yunfan and Liu, Yu-Shen and Shi, Kanle and Han, Zhizhong},
      journal={IEEE Transactions on Image Processing},
      volume={32},
      pages={2703--2718},
      year={2023},
      publisher={IEEE}
    }