VDN-NeRF

We propose VDN-NeRF, a method to train neural radiance fields (NeRF) for better geometry under non-Lambertian and dynamic lighting conditions that cause significant variations in the radiance of a point when viewed from different angles.

Paper | Data

This is the official repo for the implementation of VDN-NeRF: Resolving Shape-Radiance Ambiguity via View-Dependence Normalization.

Dependencies

torch==1.8.0
opencv_python==4.5.2.52
trimesh==3.9.8
numpy==1.19.2
pyhocon==0.3.57
icecream==2.1.0
tqdm==4.50.2
scipy==1.7.0
PyMCubes==0.1.2

Running

Training without depth feature

python dpt_runner.py --mode train --conf ./confs/womask.conf --case <case_name> -d <image_dir>

Training with depth feature

python dpt_runner.py --mode train --conf ./confs/womask_wdepth.conf --case <case_name> -d <image_dir>

Extract surface from trained model

python dpt_runner.py --mode validate_mesh --conf <config_file> --case <case_name> -d <image_dir> --is_continue # use latest checkpoint

Generate depth feature map for finetuning distillation network

python dpt_runner.py --mode getfeats_<epoch> --conf <config_file> --case <case_name> -d <image_dir> # use a specific checkpoint at epoch <epoch>

The features projected from SDF network can be found in <dataset_dir>/<case_name>/<image_dir>/depth_from_sdf

Depth features

Extract depth features from input images

cd wavelet
python predict.py --use_wavelets --normalize_input -ckpt <pre-trained checkpoint folder> -d <image_root> [-full]

Extracted features can be found in <image_root>/wavelet_feats[_full]

Finetune the distillation network

cd wavelet
python finetune_for_vdn.py --use_wavelets --normalize_input -ckpt <pre-trained checkpoint folder> -r <dataset_root> --case <case_name> -d <image_dir> -max <feature_max>

Here feature_max is decided by the distribution of depth_feature_map generated from trained VDN-NeRF, for latter use of normalize all the features to [0, 255]

Citation

Cite as below if you find this repository is helpful to your project:

@article{zhu2023vdn,
  title={VDN-NeRF: Resolving Shape-Radiance Ambiguity via View-Dependence Normalization},
  author={Zhu, Bingfan and Yang, Yanchao and Wang, Xulong and Zheng, Youyi and Guibas, Leonidas},
  journal={arXiv preprint arXiv:2303.17968},
  year={2023}
}

Acknowledgement

Some code snippets are borrowed from NeuS and WaveletMonoDepth. Thanks for these great projects.