PDFNet: Pyramid Deep Fusion Network for Two-Hand Reconstruction from RGB-D Images

Introduction

This repo is official PyTorch implementation of Pyramid Deep Fusion Network for Two-Hand Reconstruction from RGB-D Images.
Accepted by IEEE Transactions on Circuits and Systems for Video Technology(T-CSVT).

Overview of the proposed framework

Visual comparison of RGB only and RGB-D input.

Install

Environment

    conda create -n PDFNet python=3.7
    conda activate PDFNet

    # If you failed to install pytorch, you may try to modify your conda source: https://mirrors.tuna.tsinghua.edu.cn/help/anaconda/
    conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge
    
    # install pytorch3d from source if you are not using latest pytorch version
    conda install -c fvcore -c iopath -c conda-forge fvcore iopath
    conda install -c bottler nvidiacub
    conda install pytorch3d -c pytorch3d

    pip install -r requirements.txt

Directory

Root

The ${ROOT} is described as below.

${ROOT}
|-- data
|-- lib
|-- outputs
|-- scripts
|-- assets

data contains packaged dataset loaders and soft links to the datasets' image and annotation directories.
lib contains main codes for SMHR, including dataset loader code, network model, training code and other utils files.
scripts contains running scripts.
outputs contains log, trained models, imgs, and pretrained models for model outputs.
assets contains demo images.

Data

You need to follow directory structure of the data as below. (recommend to use soft links to access the datasets)

${ROOT}
|-- data
|   |-- H2O
|   |   |-- ego_view
|   |   |   |-- subject*
|   |   |-- label_split
|   |-- H2O3D
|   |   |-- evaluation
|   |   |-- train
|   |   |***.txt
|   |***.pth 
|   |***.pkl

Download the H2O dataset from the [website]
Download the H2O3D dataset from the [website]
Download pre-trained models and dataset loaders here [cloud] (Extraction code: 83di) and put them in the data folder.

Demo on two-hand samples

Prepare RGB-D image pairs into assets.
Modify demo.py to use images from H2O dataset. #L100: base_dir = 'assets/H2O/color'
run bash scripts/demo.sh
You can see rendered outputs in outputs/color/.

Training and Evaluation

Modify scripts/train.sh to load_model and use gpu.
run bash scripts/train.sh
Switch to evaluation by modify #L8: mode='train' # train, test, val
You can see rendered outputs in outputs/color/ and accuracy logs in 'H2O-val.txt'.

Outputs of images from H2O test set

Acknowledgement

The pytorch implementation of PointNet++ is based on Hand-Pointnet. The GCN network is based on IntagHand. We thank the authors for their great job!

Citation

If you find the code useful in your research, please consider citing the paper.

@article{RenPDFNet,
title={Pyramid Deep Fusion Network for Two-Hand Reconstruction from RGB-D Images},
author={Jinwei Ren, Jianke Zhu},
booktitle={IEEE Transactions on Circuits and Systems for Video Technology},
year={2024},
}

zijinxuxu/PDFNet