NSF: Neural Surface Field for Human Modeling from Monocular Depth

Project Page | Video | Paper

In ICCV 2023, Paris

Yuxuan Xue^1,, Bharat Lal Bhatnagar^2,, Riccardo Marin¹, Nikolaos Sarafianos², Yuanlu Xu², Gerard Pons-Moll¹, Tony Tung²

¹Real Virtual Human Group @ University of Tübingen & Tübingen AI Center & Max Planck Institute for Informatics
²Meta Reality Lab Research

News 🚩

[2023/08/30] NSF paper is available on ArXiv.
[2023/08/29] Code for NSF is available.
[2023/07/14] NSF is accepted to ICCV 2023, Paris.

Key Insight 🙌

Define neural fields on top of continuous surface
Learned NSF generalizes to arbitrary resolution or topology of the surface
Why benefitial For clothed human modelling:
- eliminates need for Marching Cube / Poisson Reconstruction per frame => efficient
- can reconstruct / animate human mesh resolution / topology => flexible
- keeps mesh coherency across different frames => modelling

Instruction 📘

A. Preparation

1. Dependencies

Please refer to Dependencies for:

install conda environment and required packages
obtain SMPL model
obtain prediffused SMPL skinning weights field

2. Data

Please refer to Data for:

render depth frames from scan
preprocess data

3. Path

Please specify ROOT_DIR and DATA_DIR here

B. Running

Learn Fusion Shape via SDF

Please refer to Fusion Shape for:

learn implicit fusion shape
fit SMPL-D to fusion shape
project off-surface points onto fusion shape

Learn Neural Surface Field based on Fusion Shape

Please refer to NSF for:

learn NSF to model clothed avatar
infer clothed avatar at arbitrary resolution
animate clothed avatar with desired pose sequences

C. Pre-trained Model

Please refer to Pretrained Models for:

available pretrained models for subjects in BuFF/CAPE dataset

Citation ✍️

@inproceedings{xue2023nsf,
  title     = {{NSF: Neural Surface Field for Human Modeling from Monocular Depth}},
  author    = {Xue, Yuxuan and Bhatnagar, Bharat Lal and Marin, Riccardo and Sarafianos, Nikolaos and Xu, Yuanlu and Pons-Moll, Gerard and Tung, Tony.},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  month     = {October},
  year      = {2023},
}

chuanfushen/NeuralSurfaceField