SAILOR: A Python repository from zdlarr

SAILOR: Synergizing Radiance and Occupancy Fields for Live Human Performance Capture

Zheng Dong, Xu Ke, Yaoan Gao, Qilin Sun, Hujun Bao, Weiwei Xu*, Rynson W.H. Lau
SIGGRAPH ASIA 2023

SAILOR is a generalizable method for human free-view rendering and reconstruction from very sparse (e.g., 4) RGBD streams , achieving near real-time performance under acceleration.

Our free-view rendering results and bullet-time effects on our real-captured dataset (Unseen performers ).

Installation

Please install python dependencies in requirements.txt:

conda create -n SAILOR python=3.8
conda activate SAILOR
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt

Install the surface localization algorithm named ImplicitSeg provided by MonoPort, for our fast post-merging operation.

Our code has been tested under the following system :

Ubuntu 18.04, 20.04 or 22.04
Python 3.8 and PyTorch 1.8.0
GCC/G++ 9.5.0
Nvidia GPU (RTX 3090) CUDA 11.1 CuDNN

Build the c++ and CUDA libraries:

VoxelEncoding, FastNerf and Mesh-RenderUtil: cd c_lib/* && python setup.py install. VoxelEncoding provides the CUDA accelerated versions of TSDF-Fusion, two-layer tree construction, ray-voxel intersection, adaptive points sampling, etc. FastNerf provides a fully-fused version of the MLPs and Hydra-attention for our SRONet.
AugDepth and Depth2Color [optional] (Eigen3, OpenCV, OpenMp and pybind11 are required):

cd c_lib/* 
mkdir build && cd build
cmake .. && make

Setup

Clone or download this repo
Download our pretrained depth denoising model (latest_model_BodyDRM2.pth) and our rendering model (latest_model_BasicRenNet.pth) here
Move the downloaded models to ./checkpoints_rend/SAILOR folder

Usage

The example static test data is provided in ./test_data, the data structure of static (or dynamic) is listed as :

<dataset_name>
|-- COLOR
    |-- FRAMExxxx
        |-- 0.jpg       # input RGB image (1024x1024) for each view
        |-- 1.jpg
        ...
|-- DEPTH
    |-- FRAMExxxx
        |-- 0.png       # input depth image (1024x1024, uint16, unit is m after dividing by 10000) for each view
        |-- 1.png
        ...
|-- MASK
    |-- FRAMExxxx
        |-- 0.png       # input human-region mask (1024x1024) for each view
        |-- 1.png
        ...
|-- PARAM
    |-- FRAMExxxx
        |-- 0.npy       # camera intrinsic ('K': 3x3) and pose ('RT': 3x4) matrices for each view
        |-- 1.npy

Depth denoising :

Run python -m depth_denoising.inference
The original and denoised point clouds are in the ./depth_denoising/results folder. Use meshlab to visualize the 3D results
Modify basic_path, frame idx and view_id in the file inference.py to obtain the results of other examples

SRONet and SRONetUp :

For provided static data, run python -m upsampling.inference_static --name SAILOR (in 1K resolution) or python -m SRONet.inference_static --name SAILOR (in 512 resolution), to obtain the reconstructed 3D mesh and free-view rendering results.
The reconstructed 3D meshes are in the ./checkpoints_rend/SAILOR/val_results folder. To render the 3D mesh, run python -m utils_render.render_mesh to obtain the free-view mesh rendering results. Modify opts.ren_data_root, obj_path and obj_name in the file render_mesh.py to get new results.
For dynamic data, first download our real-captured data here, unzip the data and put them in the ./test_data folder
For dynamic data, then run python -m upsampling.inference_dynamic --name SAILOR or python -m SRONet.inference_dynamic --name SAILOR to obtain the rendering results.
Modify opts.ren_data_root and opts.data_name in inference_static.py and inference_dynamic.py to obtain new rendering results
The rendering images and videos are in the ./SRONet(or upsampling)/results folder.

Interactive rendering :

We release our interactive rendering GUI for our real-captured dataset.

TensorRT is required to accelerate our depth denoising network and the encoders in SRONet(upsampling). Please refer to TensorRT installation guide and then install torch2trt. Our TensorRT version is 7.2
Run python -m depth_denoising.toTensorRT, python -m SRONet.toTensorRT and python -m upsampling.toTensorRT to obtain the TRTModules (the param opts.num_gpus in toTensorRT.py controls the number of GPUs). The final pth models are in the ./SAILOR/accelerated_models folder
Run python -m gui.gui_render. Modify the opts.ren_data_root in gui_render.py to test other data, and modify the opts.num_gpus to use 1 GPU (slow) or 2 GPUs. The GIF below shows the rendering result of using 2 Nvidia RTX 3090, an Intel i9-13900k, and an MSI Z790 god-like motherboard

License

The code, models, and GUI demos in this repository are released under the GPL-3.0 license.

Citation

If you find our work helpful to your research, please cite our paper.

@article{dong2023sailor,
author = {Zheng Dong, Xu Ke, Yaoan Gao, Qilin Sun, Hujun Bao, Weiwei Xu, Rynson W.H. Lau},
title = {SAILOR: Synergizing Radiance and Occupancy Fields for Live Human Performance Capture},
year = {2023},
journal = {ACM Transactions on Graphics (TOG)},
volume = {42},
number = {6},
doi = {10.1145/3618370},
publisher = {ACM}
}