/PersonalizedHandMeshEstimation

[ECCV2022] "Identity-Aware Hand Mesh Estimation and Personalization from RGB Images".

Primary LanguagePythonMIT LicenseMIT

Identity-Aware Hand Mesh Estimation and Personalization from RGB Images

[ECCV2022] "Identity-Aware Hand Mesh Estimation and Personalization from RGB Images".

Introduction

This repo is the PyTorch implementation of ECCV2022 paper "Identity-Aware Hand Mesh Estimation and Personalization from RGB Images". You can find this paper from paper link.

Install

  • Environment for IDentity-Aware Hand Mesh Estimation (IdHandMesh)

    conda create -n IdHandMesh python=3.8   
    conda activate IdHandMesh
    
  • Please follow official suggestions to install pytorch and torchvision. We use pytorch=1.7.1, torchvision=0.8.2

  • Requirements

    Please see environment.yaml.

  • Install MANO from MANO.

  • Note that torch_sparse and MPI-IS Mesh are required for the CMR model (one of the methods our method is compared to).

    a) If you have difficulty in installing torch_sparse etc., please use whl file from here.

    b) MPI-IS Mesh: We suggest to install this library from the source.

    More details can be found at CMR.

Dataset

Dex_YCB

  • Please download DEX-YCB dataset from this link, and create a soft link in data, i.e., data/dex_ycb. Inside the folder 'datasets', run the following scripts to process the data.
    python split_annotations_with_cropping.py
    python get_rest_hand_mesh.py
    
  • After processing the data, you should have the following directory struction.
    ${ROOT} 
    |-- conv
    |   |-- ...
    |-- data
    |   |-- dex_ycb
    |   |   |-- 20200709-subject-01
    |   |   |-- 20200813-subject-02
    |   |   |-- ...
    |   |   |-- 20201022-subject-10
    |   |   |-- 20201022-subject-10
    |   |   |-- 20201022-subject-10
    |   |   |-- bop
    |   |   |-- calibration
    |   |   |-- cropped_hand_size_2         # generated by our .py file
    |   |   |-- models
    |   |   |-- split_annotations           # generated by our .py file
    |-- datasets
    |   |-- dex_ycb
    |   |   |-- ...
    |-- options
    |-- out
    |-- scripts
    |-- src
    |-- template
    |   |-- dex_ycb_j_regressor.npy
    |   |-- MANO_RIGHT.pkl
    |   |-- template.ply 
    |   |-- transform.pkl
    |-- utils
    |   |-- ...
    |-- ....py
    

Reproducing the baseline and our method.

  • Training

    a) Train the baseline model.

    ./scripts/train_dex_ycb_mano_based_baseline.sh
    

    b) Train the our model with ground truth hand shape.

    ./scripts/train_dex_ycb_mano_based_our_model_with_gt_shape.sh
    

    c) Train baseline with confidence branch. (only train the confidence branch, other parts are frozen).

    ./scripts/train_dex_ycb_mano_based_conf_branch.sh
    
  • Run hand shape calibration.

    a) Get results from the baseline model

    python mis_dex_ycb_get_predictions_baseline_with_conf.py
    

    b) Perform calibration

    python calibrate_from_shape_params.py
    
  • Evaluate the performance without optimizatin module.

    a) Baseline performance

    ./scripts/eval_dex_ycb_mano_based_baseline.sh
    

    b) Our model when fed with groundtruth hand shape

    ./scripts/eval_dex_ycb_ours_gt_hand_shape.sh
    

    c) Our model when fed with calibrated hand shape

    ./scripts/eval_dex_ycb_ours_calibrated_hand_shape.sh
    
  • Optimizatin module during inference.

    Get 2d predictions.

    python mis_dex_ycb_get_predictions_2d.py
    

    Run optimization and evaluate at the same time.

    a) Baseline performance

    python optimization_dex_ycb_baseline.py
    

    b) Our model when fed with groundtruth hand shape

    python optimization_dex_ycb_ours_with_gt_hand_shape.py
    

    c) Our model when fed with calibrated hand shape

    python optimization_dex_ycb_ours_with_calibrated_hand_shape.py
    

Reproducing the methods being compared to.

  • CMR

    ./scripts/train_dex_ycb_cmr.sh
    ./scripts/eval_dex_ycb_cmr.sh
    
  • Boukhayma’s model

    ./scripts/train_dex_ycb_boukhayma
    ./scripts/eval_dex_ycb_boukhayma.sh
    
  • Metro

    Install Metro and then use the following script to generate the tsv file for DEX_YCB.

    metro_utils/tsv_demo_dex_ycb.py
    

Reference

@inproceedings{kong2022identity,
title={Identity-Aware Hand Mesh Estimation and Personalization from RGB Images},
author={Kong, Deying and Zhang, Linguang and Chen, Liangjian and Ma, Haoyu and Yan, Xiangyi and Sun, Shanlin and Liu, Xingwei and Han, Kun and Xie, Xiaohui},
booktitle={Computer Vision--ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part V},
pages={536--553},
year={2022}
}

Acknowledgement

Our implementation is developed with the help of the following open sourced projects:

Please also consider cite the above projects if you find them helpful.