DeepHandMesh: A Weakly-Supervised Deep Encoder-Decoder Framework for High-Fidelity Hand Mesh Modeling

Introduction

This repo is official PyTorch implementation of DeepHandMesh: A Weakly-Supervised Deep Encoder-Decoder Framework for High-Fidelity Hand Mesh Modeling (ECCV 2020. Oral.).

Demo

Download pre-trained DeepHandMesh from here.
Place the downloaded file at demo/subject_${SUBJECT_IDX} folder, where the filename is snapshot_${EPOCH}.pth.tar.
Download hand model from here and place it at data folder.
Set hand joint Euler angles at here.
Run python demo.py --gpu 0 --subject ${SUBJECT_IDX} --test_epoch ${EPOCH}.

DeepHandMesh dataset

For the DeepHandMesh dataset download and instructions, go to [HOMEPAGE].
Belows are instructions for DeepHandMesh for the weakly-supervised high-fidelity 3D hand mesh modeling.

Directory

Root

The ${ROOT} is described as below.

${ROOT}
|-- data
|-- common
|-- main
|-- output
|-- demo

data contains data loading codes and soft links to images and annotations directories.
common contains kernel codes.
main contains high-level codes for training or testing the network.
output contains log, trained models, visualized outputs, and test result.
demo contains demo codes.

Data

You need to follow directory structure of the data as below.

${ROOT}
|-- data
|   |-- images
|   |   |-- subject_1
|   |   |-- subject_2
|   |   |-- subject_3
|   |   |-- subject_4
|   |-- annotations
|   |   |-- 3D_scans_decimated
|   |   |   |-- subject_4
|   |   |-- depthmaps
|   |   |   |-- subject_4
|   |   |-- keypoints
|   |   |   |-- subject_4
|   |   |-- KRT_512
|   |-- hand_model
|   |   |-- global_pose.txt
|   |   |-- global_pose_inv.txt
|   |   |-- hand.fbx
|   |   |-- hand.obj
|   |   |-- local_pose.txt
|   |   |-- skeleton.txt
|   |   |-- skinning_weight.txt

Download datasets and hand model from [HOMEPAGE].

Output

You need to follow the directory structure of the output folder as below.

${ROOT}
|-- output
|   |-- log
|   |-- model_dump
|   |-- result
|   |-- vis

log folder contains training log file.
model_dump folder contains saved checkpoints for each epoch.
result folder contains final estimation files generated in the testing stage.
vis folder contains visualized results.

Running DeepHandMesh

Prerequisites

For the training, install neural renderer from here.
After the install, uncomment line 12 of main/model.py (from nets.DiffableRenderer.DiffableRenderer import RenderLayer) and line 40 of main/model.py (self.renderer = RenderLayer()).
If you want only testing, you do not have to install it.

Start

In the main/config.py, you can change settings of the model

Train

In the main folder, run

python train.py --gpu 0-3 --subject 4

to train the network on the GPU 0,1,2,3. --gpu 0,1,2,3 can be used instead of --gpu 0-3. You can use --continue to resume the training. Only subject 4 is supported for the training.

Test

Place trained model at the output/model_dump/subject_${SUBJECT_IDX}.

In the main folder, run

python test.py --gpu 0-3 --test_epoch 4 --subject 4

to test the network on the GPU 0,1,2,3 with snapshot_4.pth.tar. --gpu 0,1,2,3 can be used instead of --gpu 0-3.
Only subject 4 is supported for the testing. It will save images and output meshes.

Results

Here I report results of DeepHandMesh and pre-trained DeepHandMesh.

Pre-trained DeepHandMesh

Pre-trained DeepHandMesh [Download]

Effect of Identity- and Pose-Dependent Correctives

Comparison with MANO

Reference

@InProceedings{Moon_2020_ECCV_DeepHandMesh,  
author = {Moon, Gyeongsik and Shiratori, Takaaki and Lee, Kyoung Mu},  
title = {DeepHandMesh: A Weakly-supervised Deep Encoder-Decoder Framework for High-fidelity Hand Mesh Modeling},  
booktitle = {European Conference on Computer Vision (ECCV)},  
year = {2020}  
}

License

DeepHandMesh is CC-BY-NC 4.0 licensed, as found in the LICENSE file.

LiYunJamesPhD/DeepHandMesh