NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction

[Project] [Paper]

Yiming Wang*, Qin Han*, Marc Habermann, Kostas Daniilidis, Christian Theobalt, Lingjie Liu

ICCV 2023

NeuS2 is a method for fast neural surface reconstruction, which achieves two orders of magnitude improvement in terms of acceleration without compromising reconstruction quality, compared to NeuS. To accelerate the training process, we integrate multi-resolution hash encodings into a neural surface representation and implement our whole algorithm in CUDA. In addition, we extend our method for reconstructing dynamic scenes with an incremental training strategy.

This project is an extension of Instant-NGP enabling it to model neural surface representation and dynmaic scenes. We extended:

dependencies/neus2_TCNN
- add second-order derivative backpropagation computation for MLP;
- add progressive training for Grid Encoding.
neural-graphics-primitives
- extend NeRF mode for NeuS;
- add support for dynamic scenes.

Updates

[08/30/2023] Released NeuS2++ (See neuspp branch). Now you can reconstruct unmasked/unbounded scenes in seconds!
[08/15/2023] Released official codes!

Gallery
Installation
Training
Data
Data Convention
Acknowledgements & Citation

Gallery


Synthetic Scene surface reconstruction in 5 minutes Input: multi-view images with mask Output: freeview synthesis, mesh static-scene-lego-5min.mp4	Synthetic Scene surface reconstruction in 5 minutes Comparison between NeuS (8h), Instant-NGP (5min) and NeuS2 (5min) Download dataset from Google Drive static-scene-dtu-scan20-compressed.mov
Dynamic Scene surface reconstruction in 20s per frame Input: a sequence of multi-view images with mask Output: novel view synthesis, mesh dynamic-scene-dynacap.mp4	Long sequence surface reconstruction with 2000 frames Input: a sequence of multi-view images with mask NeuS2 can handle long sequences input with large movements long_sequences_compressed.mp4

Installation

Please first see Instant-NGP for original requirements and compilation instructions. NeuS2 follows the installing steps of Instant-NGP.

For CRC:

module load cmake/3.26.4

Install PyTorch3D:

conda create -n pytorch3d python=3.9
conda activate pytorch3d
conda install -c "nvidia/label/cuda-11.6.2" cuda-toolkit
conda install pytorch=1.13.0 torchvision pytorch-cuda=11.6 -c pytorch -c nvidia
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install pytorch3d -c pytorch3d
python NeuS2/test_pytorch3d.py

Clone this repository and all its submodules using the following command:

git clone --recursive https://github.com/19reborn/NeuS2
cd NeuS2

Then use CMake to build the project:

cmake . -B build
cmake --build build --config RelWithDebInfo -j 16

For python useage, first install dependencies with conda and pip:

conda create -n neus2 python=3.9
conda activate neus2
pip install -r requirements.txt

Then install pytorch and pytorch3d.

If you meet problems of compiling, you may find solutions here.

Training

Static Scene

You can specify a static scene by setting --scene to a .json file containing data descriptions.

The DTU Scan24 scene can be downloaded from Google Drive:

./build/testbed --scene ${data_path}/transform.json

Or, you can run the experiment in an automated fashion through python bindings:

python scripts/run.py --scene ${data_path}/transform.json --name ${your_experiment_name} --network ${config_name} --n_steps ${training_steps}

For training on DTU dataset, config_name can be dtu.json and training_steps can be 15000.

Also, you can use scripts/run_dynamic.py as:

python scripts/run_dynamic.py --scene ${data_path}/transform.json --name ${your_experiment_name} --network ${config_name}

, where the number of training iterations is specified in the config.

The outputs and logs of the experiment can be found at output/${your_experiment_name}/.

Dynamic Scene

To specify a dynamic scene, you should set --scene to a directory containing .json files that describe training frames.

./build/testbed --scene ${data_dirname}

Or, run scripts/run_dynamic.py using python:

python scripts/run_dynamic.py --scene ${data_dirname} --name ${your_experiment_name} --network ${config_name}

There are some hyperparameters of the network configuration, such as configs/nerf/base.json, to control the dynamic training process:

first_frame_max_training_step: determine the number of training iterations for the first frame, default 2000.
next_frame_max_training_step: determine the number of training iterations for subsequent frames, default 1300, including global transformation prediction.
predict_global_movement: set true if use global transformation prediction.
predict_global_movement_training_step: determine the number of training iterations for global transformation prediction, default 300. Only valid when predict_global_movement is true.

Also, we provide scripts to reconstruct dynamic scenes by reconstructing static scene frame by frame.

python scripts/run_per_frame.py --base_dir ${data_dirname} --output_dir ${output_path} --config ${config_name}

Dynamic scene examples can be downloaded from Google Drive.

Data

Static scene example, link.
Dynamic scene example, link.
Pretrained model and configuration files for DTU, link.

Data Convention

NeuS2 supports the data format provided by Instant-NGP. Also, you can use NeuS2's data format (with from_na=true), see data convention.

We also provide a data conversion from NeuS to our data convention, which can be found in tools/data_format_from_neus.py.

Acknowledgements & Citation

NeuS2

@inproceedings{neus2,
    title={NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction}, 
    author={Wang, Yiming and Han, Qin and Habermann, Marc and Daniilidis, Kostas and Theobalt, Christian and Liu, Lingjie},
    year={2023},
    booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}
}

Instant-NGP

@article{mueller2022instant,
    author = {Thomas M\"uller and Alex Evans and Christoph Schied and Alexander Keller},
    title = {Instant Neural Graphics Primitives with a Multiresolution Hash Encoding},
    journal = {ACM Trans. Graph.},
    issue_date = {July 2022},
    volume = {41},
    number = {4},
    month = jul,
    year = {2022},
    pages = {102:1--102:15},
    articleno = {102},
    numpages = {15},
    url = {https://doi.org/10.1145/3528223.3530127},
    doi = {10.1145/3528223.3530127},
    publisher = {ACM},
    address = {New York, NY, USA},
}

NeuS

@inproceedings{wang2021neus,
	title={NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction},
	author={Wang, Peng and Liu, Lingjie and Liu, Yuan and Theobalt, Christian and Komura, Taku and Wang, Wenping},
	booktitle={Proc. Advances in Neural Information Processing Systems (NeurIPS)},
	volume={34},
	pages={27171--27183},
	year={2021}
}

JCBreath/NeuS2