PlenVDB: Memory Efficient VDB-Based Radiance Fields for Fast Training and Rendering (CVPR2023)

project page | paper | video

Code release for the CVPR 2023 paper "PlenVDB: Memory Efficient VDB-Based Radiance Fields for Fast Training and Rendering"

Han Yan, Celong Liu, Chao Ma, Xing Mei

Visualization

Download pretrained DensityVDB and then you can check the geometry of the objects in the Blender.

News

[2023/05/29] Code initial release v1.0
[2023/02/28] PlenVDB is accepted by CVPR 2023.

File Tree

File tree (click to expand)

.PlenVDB  
├── doc  
│   └── additional_cmds.md  
├── figs  
│   └── teaser.png  
├── openvdb  
│   └── ...  
├── plenvdb  
│   ├── configs  
│   │   └── ...  
│   ├── cuda  
│   ├── lib  
│   │   ├── ...  
│   │   └── vdb  
│   │       ├── CMakeLists.txt  
│   │       ├── colorvdb.cu  
│   │       ├── densityvdb.cu  
│   │       ├── plenvdb.cpp  
│   │       ├── plenvdb.cu  
│   │       ├── plenvdb.cuh  
│   │       ├── plenvdb.h  
│   │       └── renderer.cu  
│   ├── LICENSE  
│   ├── requirements.txt  
│   ├── run.py  
│   ├── tools  
│   └── vdb_compression.py  
└── README.md

Installation

Requirements

Ubuntu, python==3.7, g++>=6.3.1, gcc>=6.3.1, CMake>=3.18.0, Boost>=1.70, TBB>=2019.0, Blosc>=1.7.0
pytorch and torch-scatter is dependent on CUDA, please install the correct version for your machine

1. Compile OpenVDB, NanoVDB, PyOpenVDB

First, let's compile OpenVDB, NanoVDB and the python module. We mainly focus on PlenVDB/openvdb directory, which is an old version of OpenVDB library. We have tested on g++7.5.0, gcc7.5.0, make3.18.0,libboost1.74 tbb2019 and libblosc1.7.0. And you can go to PlenVDB/openvdb/cmake/config/OpenVDBVersions.cmake for detailed version requirements about the dependencies. If you have some trouble with the dependencies, we hope the commands in the Additional Commands will help.

When all dependencies are ready, run

cd PlenVDB/openvdb
mkdir build
cd build
cmake -DOPENVDB_BUILD_NANOVDB=ON -DOPENVDB_BUILD_PYTHON_MODULE=ON -DUSE_NUMPY=ON ..
sudo make -j4
sudo make install

2. Env Setting

Second, let's create an environment for running. Here we give the CUDA10.2 version.

# cd PlenVDB/plenvdb/
conda create -n plenvdb python=3.7
conda activate plenvdb
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=10.2 -c pytorch
pip install -r requirements.txt
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple/  pytest
pip install torch-scatter -f https://data.pyg.org/whl/torch-1.10.1+cu102.html
pip install imageio-ffmpeg

3. Compile PlenVDB

Third, let's compile the plenvdb.so, which is a VDB-based data structure.

cd PlenVDB/plenvdb/lib/vdb/
mkdir build
cd build
cmake ..
make -j4

If you meet with some difficulties in compilation, I recommende you to read its repository or document.

Prepare Datasets

Download NeRF-Synthetic, NSVF, BlendedMVS, DeepVoxels.

Modify configurations in PlenVDB/plenvdb/configs to indicate the datadir.

Training

Run for mic and get finedensity.vdb and finecolor.vdb

cd PlenVDB/plenvdb/
python run.py --config configs/nerf/mic.py

Merge two VDBs into one: mergedidxs.vdb and mergeddata.npz

python vdb_compression.py --basedir logs/nerf_synthetic/ --scenes mic

Render video with merged VDB

python run.py --config configs/nerf/mic.py --render_video --render_only --use_mergedvdb

TODO List

Update the dependency to OpenVDB10.0.2 (OpenVDB9.1.1 at now).
Support renderer for face-forward and unbounded scenes.
Release code for rendering VDB on multiply platforms, e.g. iOS, Android, etc.
Support pytorch extension.

Citation

Please cite the following paper if you use this repository in your reseach.

@inproceedings{hyan2023plenvdb,
      title={PlenVDB: Memory Efficient VDB-Based Radiance Fields for Fast Training and Rendering },
      author={Han Yan and Celong Liu and Chao Ma and Xing Mei},
      year={2023},
      booktitle={CVPR},
}

Acknowledgement

This code is built upon the publicly available code DVGO and OpenVDB. Thanks the authors of DVGO and OpenVDB for making their excellent work and codes publicly available.

wolfball/PlenVDB