HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations

HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations
Peng Dai, Yang Zhang, Tao Liu, Zhen Fan, Tianyuan Du, Zhuo Su, Xiaozheng Zheng, Zeming Li
PICO, ByteDance
🥳 Accepted to CVPR 2024

[arXiv] [Project]

📣 Updates

Release the pre-trained models and the evaluation results. (Expected to be ready in 03/2024)
Release the PICO-FreeDancing dataset.
Release the training and testing codes.

🖥️ Requirements

Python >= 3.9
PyTorch >= 2.0.1
numpy >= 1.23.1
human_body_prior

⚒️ Preparation

AMASS

Please download the datasets from AMASS and place them in ./data/AMASS directory of this repository.
Download the required body models and place them in ./body_models directory of this repository. For the SMPL+H body model, download it from http://mano.is.tue.mpg.de/. Please download the AMASS version of the model with DMPL blendshapes. You can obtain dynamic shape blendshapes, e.g. DMPLs, from http://smpl.is.tue.mpg.de.
Run ./prepare_data.py to preprocess the input data for faster training. The data split for training and testing data under Protocol 1 in our paper is stored under the folder ./prepare_data/data_split (directly copy from AvatarPoser).

python ./prepare_data.py --support_dir ./body_models/ --root_dir ./data/AMASS/ --save_dir [path_to_save]

🚴 Training

Modify the dataset_path in ./options/train_config.yaml to your [path_to_save].

python train.py --config ./options/train_config.yaml

🏃‍♀️ Evaluation

Modify the resume_model path in ./options/test_config.yaml.

python test.py --config ./options/test_config.yaml

🍭 Pre-trained Model

Protocol1

Trained Model: coming soon.

Input Type	MPJRE	MPJPE	MPJVE	Jitter
HMD	-	-	-	-
HMD+2IMUs	-	-	-	-
HMD+3IMUs	-	-	-	-

🎉 PICO-FreeDancing Dataset

Brief description of the dataset

There are in total 74 free-dancing motions from 8 subjects (3 male and 5 female).

For each motion, there are two files: gt_body_parms.pt and hmd_sensor_data.pt.

gt_body_parms.pt contains the ground-truth SMPL parameters obtained via OptiTrack and Mosh++.

hmd_sensor_data.pt contains the synchronized real-captured HMD and IMU sensor data. Specifically, it has three types of data:

sensor_coordinates: with a shape of N * [head, left_hand, right_hand] * 3,
sensor_orientation: with a shape of N * [head, left_hand, right_hand, left_foot, right_foot] * 3 * 3
sensor_acceleration: with a shape of N * [head, left_hand, right_hand, left_foot, right_foot] * 3

where N is the number of frames.

Download

Google Drive

🤟 Citation

If you find our work useful for your research, please consider citing the paper:

@inproceedings{
  daip2024hmdposer,
  title={HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations},
  author={Dai, Peng and Zhang, Yang and Liu, Tao and Fan, Zhen and Du, Tianyuan and Su, Zhuo and Zheng, Xiaozheng and Li, Zeming},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2024}
}

🗞️ License

Distributed under the MIT License. See LICENSE for more information.

🙌 Acknowledgements

This project refers to source codes shared by AvatarPoser. We thank the authors for their great job!

Pico-AI-Team/HMD-Poser