wen-yuan-zhang/Fast-Learning-NeRF

[TIP'2023] Fast Learning Radiance Fields by Shooting Much Fewer Rays, a general strategy to speed up the learning of radiance field

Fast Learning Radiance Fields by Shooting Much Fewer Rays

Wenyuan Zhang¹, Ruofan Xing¹, Yunfan Zeng¹, Yu-Shen Liu¹, Kanle Shi², Zhizhong Han³

¹Tsinghua University, ²Kuaishou Technology, ³Wayne State University

Project Page | Paper

In this work, we introduce a general strategy to speed up the learning procedure for almost all radiance fields based methods by shooting much fewer rays.

Setup

Python 3 dependencies:

• pytorch
• matplotlib
• numpy
• imageio
• configargparse

Specifically, our experiments are conducted on six radiance field based methods, including NeRF, NeRF++, Plenoxels, Mip-NeRF 360, Instant-NGP and NeuS. We finetuned their official codes and apply our method to the baseline methods. To setup each individual environment for each experiment, please refer to the project pages of the baseline methods.

Data preparation

We use five datasets named synthetic dataset, llff dataset, lf dataset, tanks_and_temples dataset, which are widely used in radiance fields based methods. You can download the four datasets from the following links.

Synthetic Dataset | LF Dataset | LLFF Dataset

Tanks And Temples Dataset (for NeRF++/Plenoxels) | (for Instant-NGP)

Running code

Running NeRF-based Method:

cd nerf-ours

Training

CUDA_VISIBLE_DEVICES=0 python run_nerf.py --config configs/lego.txt

Testing

CUDA_VISIBLE_DEVICES=0 python run_nerf.py --config configs/lego.txt --ft_path logs/paper_lego/011.tar --render_only --render_test

Running NeRF++-based Method:

cd nerf++-ours

Training

CUDA_VISIBLE_DEVICES=0 python ddp_train_nerf.py --config configs/tanks_and_temples/tat_training_truck.txt

Testing

CUDA_VISIBLE_DEVICES=0 python ddp_test_nerf.py --config configs/tanks_and_temples/tat_training_truck.txt --render_splits test

Running Plenoxels-based Method:

cd plenoxels-ours/opt

Training

CUDA_VISIBLE_DEVICES=0 python opt.py data/nerf_synthetic/lego -t ckpt/paper_lego -c configs/syn.json

Testing

CUDA_VISIBLE_DEVICES=0 python render_imgs.py ckpt/paper_lego/ckpt.npz data/nerf_synthetic/lego

Running Instant-NGP-based Method:

cd ngp-ours

Training & Testing

CUDA_VISIBLE_DEVICES=0 python python main_nerf.py data/TanksAndTemple/Family --workspace log/tnt/Family -O --bound 1.0 --scale 0.33 --dt_gamma 0

Some Visualization Results

Visulization of Quadtree Subdivision Procedure

Visualization of Comparison on Synthetic Dataset

Visualization of Comparison on LF Dataset

Visualization of Comparison on Tanks And Temples Dataset

Visualization of Comparison on Mip-NeRF 360 Dataset

Visualization of Comparison on DTU Dataset

Citation

@article{zhang2023fast,
      title={Fast Learning Radiance Fields by Shooting Much Fewer Rays},
      author={Zhang, Wenyuan and Xing, Ruofan and Zeng, Yunfan and Liu, Yu-Shen and Shi, Kanle and Han, Zhizhong},
      journal={IEEE Transactions on Image Processing},
      volume={32},
      pages={2703--2718},
      year={2023},
      publisher={IEEE}
    }