SynergyNeRF

[ArXiv] [Project]

Official PyTorch implementation of SynergyNeRF, as presented in our paper:

Synergistic Integration of Coordinate Network and Tensorial Feature for Improving Neural Radiance Fields from Sparse Inputs (ICML2024)
Mingyu Kim¹, Jun-Seong Kim², Se-Young Yun^1† and Jin-Hwa Kim^3†
¹KAIST AI, ²POSTECH EE, ³NAVER AI Lab.
(^† indicates corresponding authors)

Update

Training code.
Inference code.
Datasets.

News

(24.05.01) Our paper has been accepted in ICML 2024.
(23.10.21) Our paper has been presented at NeurIPS2023 DeepInverse Workshop.

Environment Setup

# create conda environment
conda create --name SynergyNeRF python=3.9

# activate env
conda activate SynergyNeRF

# install pytorch >= 1.12 (e.g cu116)
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116

# install packages
pip install -r requirements.txt

This implementation utilizes the code bases of TensoRF and HexPlane.

Dataset

For static NeRFs, this implementation utilizes NeRF Synthetic and TankandTemples.

# NeRF Synthetic: download and extract nerf_synthetic.zip
cd SynergyNeRF_3D
mkdir data
cd data
unzip nerf_synthetic.zip 

# tankandtemples: download and extract TankAndTemple.zip
unzip TankAndTemple.zip

For dynamic NeRFs, this implementation utilizes D-NeRF Dataset.

# D-NeRF dataset: download and extract data.zip
cd SynergyNeRF_4D
mkdir data
cd data
unzip data.zip

Quick Start

We provide training scripts based on config files.
First, we illustrate training static NeRFs for nerf_synthetic and tankandtemples.
To run this code on GPUs with 24GB of VRAM, you should use the configuration files located in config/SynergyNeRF/revised_cfg/.
If you want to run the code as described in the original paper,
please use the configuration files found in config/SynergyNeRF_disentangled/official/.

# training nerf_synthetic
cd SynergyNeRF_3D 
# scenes : {chair, drums, ficus, lego, hotdog, materials, mic, ship}
python main.py --config=config/SynergyNeRF/revised_cfg/8_views/{scene}.yaml

# training tankandtemples
# scenes : {barn, caterpillar, family, truck}
python main.py --config=config/SynergyNeRF/tankandtemples_cfgs/{scene}.yaml

Second, we illustrate training dynamic NeRFs for the D-NeRF dataset.

# training nerf_synthetic
cd SynergyNeRF_4D 
# scenes : {bouncingballs, hellwarrior, hook, jumpingjacks, lego, mutant, standup, trex}
python main.py --config=config/SynergyNeRF/official/{scene}.yaml

Bibliography

@InProceedings{kim2024synergistic,
  author    = {Kim, Mingyu and Kim, Jun Seong and Yun, Se Young and Kim, Jin Hwa},  
  title     = {Synergistic Integration of Coordinate Network and Tensorial Feature for Improving NeRFs from Sparse Inputs},  
  booktitle = {Proceedings of the 41th International Conference on Machine Learning},
  year      = {2024},
  series    = {Proceedings of Machine Learning Research},
  publisher = {PMLR},  
}

Acknowledgement

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [No.2022-0-00641, XVoice: Multi-Modal Voice Meta Learning]. A portion of this work was carried out during an internship at NAVER AI Lab. We also extend our gratitude to ACTNOVA for providing the computational resources required.

MingyuKim87/SynergyNeRF