This repository contains a PyTorch implementation of our paper "Multi-task View Synthesis with Neural Radiance Fields".
To install the dependencies, follow the official repository of GeoNeRF:
The datasets are hosted on HuggingFace. "replica_training_views.zip" and "scenenet_training_views.zip" are used for training in the main experiments. "replica_training_full.zip" and "scenenet_training_full.zip" are used for held-in testing on novel views of already seen scenes. "replica_testing_full.zip" and "scenenet_testing_full.zip" are used for novel scene evaluation within the same dataset (in-distribution). "llff_full.zip", "tartan_full.zip", "scannet_full.zip", "blended_full.zip" are used for out-of-distribution scene evaluation.
Our training process contains two stages following cross-stitch networks. In the first stage, we train all the parameters except for the self-attention modules in the CTA module for 5,000 iterations. Run the command:
bash train_first_stage_scripts.sh
Afterwards, before proceeding to the second stage, change the path of the loaded pretrained weight on L764 in run_geo_nerf.py. Then run the command:
bash train_second_stage_scripts.sh
Before testing, change the path of the loaded pretrained weight on L778. Running the following scripts will start testing the multi-task view synthesis task on three novel scenes.
bash replica_test_scripts.sh
If you find our work useful, please consider citing:
@inproceedings{zheng2023mtvs,
title={Multi-task View Synthesis with Neural Radiance Fields},
author={Zheng, Shuhong and Bao, Zhipeng and Hebert, Martial and Wang, Yu-Xiong},
booktitle={IEEE/CVF International Conference on Computer Vision (ICCV)},
year={2023}
}The codes are largely borrowed from the PyTorch implementation of GeoNeRF:
https://github.com/idiap/GeoNeRF
This work was supported in part by NSF Grant 2106825, Toyota Research Institute, NIFA Award 2020-67021-32799, the Jump ARCHES endowment, the NCSA Fellows program, the Illinois-Insper Partnership, and the Amazon Research Award. This work used NVIDIA GPUs at NCSA Delta through allocations CIS220014 and CIS230012 from the ACCESS program.