NeRF4SeRe: Neural Radiance Fields for Scene Reconstruction

Project of AI3604 Computer Vision, SJTU.
Wei Jiang * · Kangrui Mao * · Gu Zhang * · Haoyu Zhen *
\star = equal contribution

TODO

installation guideline
training and testing code for NeRF, INeRF and Instant-NGP
implementation of NeRFusion
NICE-SLAM integration
visualizations

Installation

git clone --recursive https://github.com/NeRF-SeRe/SeRe.git
cd SeRe

Environment setup You should install google-sparsehash first following the instructions in torchsparse (e.g., sudo apt-get install libsparsehash-dev on Ubuntu and brew install google-sparsehash on Mac OS).

Then install the dependencies:

conda create -c conda-forge -n sere python=3.9
pip install -r requirements.txt
pip install git+https://github.com/facebookresearch/pytorch3d.git
pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git@v1.4.0
export PYTHONPATH="/path/to/sere:$PYTHONPATH"

Dataset preparation Please refer to https://github.com/cvg/nice-slam to download the Replica Dataset. For SJTUers, you could download it from Replica-JBox.

Then link the dataset
```
ln -s /path/to/Replica data/replica
```

Training Pipeline

Adding --device x to specify the GPU device (default: 0).

Vanilla NeRF

python run/main_nerf.py --config configs/nerf/replica/room0.txt

Instant-NGP

python run/main_ngp.py data/replica/room0/ --workspace output/ngp/replica/room0/ --fp16 --cuda_ray

NeRFusion

python run/main_nerfusion.py data/replica/room0 --fusion_path data/replica --workspace output/nerfusion/replica/room0 --cuda_ray --fp16 

python run/main_nerfusion.py data/replica/room0 --fusion_path data/replica --fusion_scenes office0 office1 --workspace output/nerfusion/replica/room0 --cuda_ray --fp16 # change fusion_scenes to specify the scenes to be trained

Refine camera pose (INeRF)

After you have trained a NeRF model, please put the checkpoints (its name should keep the same as model_name in config) under /ckpts/inerf. Then run:
```
python run/main_inerf.py --config configs/inerf/replica/room0.txt
```

Evaluation

If you have already trained a NeRF model, just add --test to test it. eg.

python run/main_ngp.py data/replica/room0/ --workspace output/ngp/replica/room0/ --fp16 --cuda_ray --test # write images

python run/main_ngp.py data/replica/room0/ --workspace output/ngp/replica/room0/ --fp16 --cuda_ray --test --video # generate video

Acknowledgement

For TA

Our latest report and appendices are in docs.