/NeROIC_windows

Modified NeROIC to run in windows and fully support custom LLFF datasets

Primary LanguagePythonOtherNOASSERTION

NeROIC: Neural Object Capture and Rendering from Online Image Collections

This work has been tested on Ubuntu 20.04.

This repo is a testing grounds for NeROIC on Windows 11, including dataset creation and modified datasets to be compatible with comparable repositories.

Discord Contact is in my profile for dataset help -Sazoji

quick todo: impliment direct colmap(or NeRFjson/LLFF+filename) dataloading, recursive image conversion without glob deletion, better masks, convert my nvdiffrec fork scripts to create datasets from video in a unified manner

Quick Start

  • Install required libraries.

Certain repos in this env require building from source with VS!

conda env create -f environment.yaml
conda activate neroic

apt-get install graphicsmagick-imagemagick-compat (install imagemagick to PATH for now, to be replaced with imageio)

  • Download our example data at here.

  • Optimize the geometry network.

python train.py \
--config <path_of_the_geometry_stage_config_file> \
--datadir <path of the data> 

for example:

python train.py \
--config configs/milkbox_geometry.yaml \
--datadir ./data/milkbox_dataset

(Optional) running the script with multiple GPU is a bit tricky: you should first run it a single GPU (which will generate the resized images), then stop the process before the training starts. Then run:

CUDA_VISIBLE_DEVICES=0,1,2,3 python train.py \
--config <path_of_the_geometry_stage_config_file> \
--datadir <path of the data> \
--num_gpus 4
  • Extract the normal from learned geometry.
python generate_normal.py \
 --config <path_of_the_geometry_stage_config_file> \
 --ft_path <path_of_the_geometry_network_weight_file> \
--datadir <path of the data> 

for example:

python generate_normal.py \
--config configs/milkbox_geometry.yaml \
--ft_path logs/milkbox_geometry/epoch=29.ckpt \
--datadir data/milkbox_dataset
  • Optimize the rendering network.
python train.py \
--config <path_of_the_rendering_stage_config_file> \
--ft_path <path_of_the_geometry_network_weight_file> \
--datadir <path of the data> 

for example (training with 4 GPUs):

CUDA_VISIBLE_DEVICES=0,1,2,3 python train.py \
--config configs/milkbox_rendering.yaml \
--ft_path logs/milkbox_geometry/epoch=29.ckpt \
--datadir data/milkbox_dataset \
--num_gpus 4

training logs & results will be saved at logs/.

Video Generation

the video of novel testing views is generated on-the-fly during training, and saved under logs/. To render video with a pre-trained model, run:

python train.py \
--config <path_of_the_config_file> \
--ft_path <path_of_the_model_weight_file> \
--datadir <path of the data> \
--i_video 1 \ 
(optional) --test_img_id <index_of_the_reference_image>

Testing

We provide additional testing scripts for material decomposition and relighting.

To decompose materials:

python test_material.py \
--config <path_of_the_rendering_stage_config_file> \
--ft_path <path_of_the_rendering_network_weight_file> \
--datadir <path of the data> 

To relight the model (panoramic exr HDRI maps are prefered):

python test_relighting.py \
--config <path_of_the_rendering_stage_config_file> \
--ft_path <path_of_the_rendering_network_weight_file> \
--datadir <path of the data> \
--test_env_filename <path_of_the_envrionment_map> 

Bring Your Own Data

Coming Soon

Model Overview

Our two-stage model takes images of an object from different conditions as input. With the camera poses of images and object foreground masks acquired by other state-of-the-art methods, We first optimize the geometry of scanned object and refine camera poses by training a NeRF-based network; We then compute the surface normal from the geometry (represented by density function) using our normal extraction layer; Finally, our second stage model decomposes the material properties of the object and solves for the lighting conditions for each image.

Screenshot

Novel View Synthesis

Given online images from a common object, our model can synthesize novel views of the object with the lighting conditions from the training images.

nvs.mp4

Screenshot

Material Decomposition

material.mp4

Screenshot

Relighting

relighting.mp4

Citation

If you find this useful, please cite the following:

@article{10.1145/3528223.3530177,
author = {Kuang, Zhengfei and Olszewski, Kyle and Chai, Menglei and Huang, Zeng and Achlioptas, Panos and Tulyakov, Sergey},
title = {NeROIC: Neural Rendering of Objects from Online Image Collections},
year = {2022},
issue_date = {July 2022},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {41},
number = {4},
issn = {0730-0301},
url = {https://doi.org/10.1145/3528223.3530177},
doi = {10.1145/3528223.3530177},
journal = {ACM Trans. Graph.},
month = {jul},
articleno = {56},
numpages = {12},
keywords = {neural rendering, reflectance & shading models, multi-view & 3D}
}

Acknowledgements: This work borrows many code from NeRF-pl. We thank the author for releasing his code.