Implementation of Differentiable Refraction-Tracing for Mesh Reconstruction of Transparent Objects.
Project page | paper | video
We introduce a reconstruction method for solid transparent objects. Using a static background with a coded pattern, we establish a mapping between the camera view rays and locations on the background (Environment Matting). Differentiable tracing of refractive ray paths is then used to directly optimize a 3D mesh approximation(visual hull) of the object, while simultaneously ensuring silhouette consistency and smoothness.
Requirements: (tested on ubuntu 16.04)
- NVIDIA-OptiX-SDK-6.5.0 : Used to find the triangles intersected by each ray path.
- Meshlab: We use meshlabserver to remesh and calculate average per-vertex distance (Hausdorff Distance). Note that meshlab older than 2020.04 may not support explicit remesh operation.(Tested on MeshLabServer2020.04-linux.AppImage)
- Python with pytorch, numpy, trimesh, opencv-python, h5py, tqdm and imageio.
- Ninja build system (needed by pytorch's cpp_extension). Run
apt install ninja-buildif you use ubuntu.
Our example data is captured by camera point grey or cellphone Redmi and released here
Scanned mesh(GT) and visual hull already contained in ./data/
- Download Optix-SDK, meshlabserver and our captured data.
- Set corresponding paths in
config.py. Note that you should use absolute path to Optix. - Run
python optim.py - Reconstructed mesh will be saved in ./result/ by default.
Please cite the paper in your publications if it helps your research:
@article{DRT,
title = {Differentiable Refraction-Tracing for Mesh Reconstruction of Transparent Objects},
author = {Jiahui Lyu and Bojian Wu and Dani Lischinski and Daniel Cohen-Or and Hui Huang},
journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA 2020)},
volume = {39},
number = {6},
pages = {195:1--195:13},
year = {2020},
}