Junzhe Lu, Jing Lin, Hongkun Dou, Ailing Zeng, Yue Deng, Yulun Zhang, Haoqian Wang
π An overview of DPoserβs versatility and performance across multiple pose-related tasks
Welcome to the official implementation of DPoser: Diffusion Model as Robust 3D Human Pose Prior. π
In this repository, we're excited to introduce DPoser, a robust 3D human pose prior leveraging diffusion models. DPoser is designed to enhance various pose-centric applications like human mesh recovery, pose completion, and motion denoising. Let's dive in!
We highly recommend switching to the 'v2' branch, which supports additional pose priors and features cleaner, more structured code. To switch, use the following command:
git checkout v2
-
Tested Configuration: Our code works great on PyTorch 1.12.1 with CUDA 11.3.
-
Installation Recommendation:
conda install pytorch==1.12.1 torchvision==0.13.1 cudatoolkit=11.3 -c pytorch
-
Required Python Packages:
pip install requirements.txt
-
Body Models: We use the SMPLX body model in our experiments. Make sure to set the
--bodymodel-path
parameter correctly in scripts like./run/demo.py
and./run/train.py
based on your body model's download location.
-
Pre-trained Model: Grab the pre-trained DPoser model from here and place it in
./pretrained_models
. -
Sample Data: Check out
./examples
for some sample files, including 500 body poses from the AMASS dataset and a motion sequence fragment. -
Explore DPoser Tasks:
Generate poses and save rendered images:
python -m run.demo --config configs/subvp/amass_scorefc_continuous.py --task generation
For videos of the generation process:
python -m run.demo --config configs/subvp/amass_scorefc_continuous.py --task generation_process
Complete poses and view results:
python -m run.demo --config configs/subvp/amass_scorefc_continuous.py --task completion --hypo 10 --part right_arm --view right_half
Explore other solvers like ScoreSDE for our DPoser prior:
python -m run.demo --config configs/subvp/amass_scorefc_continuous.py --task completion2 --hypo 10 --part right_arm --view right_half
Summarize visual results in a video:
python -m run.motion_denoising --config configs/subvp/amass_scorefc_continuous.py --file-path ./examples/Gestures_3_poses_batch005.npz --noise-std 0.04
Use the detected 2D keypoints from openpose and save fitting results:
python -m run.demo_fit --img=./examples/image_00077.jpg --openpose=./examples/image_00077_keypoints.json
To train DPoser, we use the AMASS dataset. You have two options for dataset preparation:
-
Option 1: Process the Dataset Yourself
Download the AMASS dataset and process it using the following script:python -m lib/data/script.py
Ensure you follow this directory structure:
${ROOT} |-- data | |-- AMASS | |-- amass_processed | |-- version1 | |-- test | |-- betas.pt | |-- pose_body.pt | |-- root_orient.pt | |-- train | |-- valid
-
Option 2: Use Preprocessed Data
Alternatively, download the processed data directly from Google Drive.
After setting up your dataset, begin training DPoser:
python -m run.train --config configs/subvp/amass_scorefc_continuous.py --name reproduce
This command will start the training process. The checkpoints, TensorBoard logs, and validation visualization results will be stored under ./output/amass_amass
.
Quantitatively evaluate 500 generated samples using this script:
python -m run.demo --config configs/subvp/amass_scorefc_continuous.py --task generation --metrics
This will use the SMPL body model to evaluate APD and SI following Pose-NDF.
For testing on the AMASS dataset (make sure you've completed the dataset preparation in Step 4):
python -m run.completion --config configs/subvp/amass_scorefc_continuous.py --gpus 1 --hypo 10 --sample 10 --part legs
To evaluate motion denoising on the AMASS dataset, use the following steps:
- Split the
HumanEva
part of the AMASS dataset into fragments using this script:python lib/dataset/HumanEva.py --input-dir path_to_HumanEva --output-dir ./data/HumanEva_60frame --seq-len 60
- Then, run this script to evaluate the motion denoising task on all sub-sequences in the
data-dir
:python -m run.motion_denoising --config configs/subvp/amass_scorefc_continuous.py --data-dir ./data/HumanEva_60frame --noise-std 0.04
To test on the EHF dataset, follow these steps:
- First, download the EHF dataset from SMPLX.
- Specify the
--data-dir
and run this script:python -m run.fitting --config configs/subvp/amass_scorefc_continuous.py --data-dir path_to_EHF --outdir ./fitting_results
-
RuntimeError: Subtraction, the '-' operator, with a bool tensor is not supported. If you are trying to invert a mask, use the '~' or 'logical_not()' operator instead.
: Solution here -
TypeError: startswith first arg must be bytes or a tuple of bytes, not str.
: Fix here.
Big thanks to ScoreSDE, GFPose, and Hand4Whole for their foundational work and code.
@article{lu2023dposer,
title={DPoser: Diffusion Model as Robust 3D Human Pose Prior},
author={Lu, Junzhe and Lin, Jing and Dou, Hongkun and Zhang, Yulun and Deng, Yue and Wang, Haoqian},
journal={arXiv preprint arXiv:2312.05541},
year={2023}
}