Official implementation of our paper:
Auxiliary Tasks Benefit 3D Skeleton-based Human Motion Prediction
Chenxin Xu, Robby T. Tan, Yuhong Tan, Siheng Chen, Xinchao Wang, Yanfeng Wang
ICCV 2023 [paper]
Abstract: Exploring spatial-temporal dependencies from observed motions is one of the core challenges of human motion pre- diction. Previous methods mainly focus on dedicated net- work structures to model the spatial and temporal dependen- cies. This paper considers a new direction by introducing a model learning framework with auxiliary tasks. In our auxiliary tasks, partial body joints’ coordinates are corrupted by either masking or adding noise and the goal is to recover corrupted coordinates depending on the rest coor- dinates. To work with auxiliary tasks, we propose a novel auxiliary-adapted transformer, which can handle incomplete, corrupted motion data and achieve coordinate recovery via capturing spatial-temporal dependencies. Through auxil- iary tasks, the auxiliary-adapted transformer is promoted to capture more comprehensive spatial-temporal dependencies among body joints’ coordinates, leading to better feature learning. Extensive experimental results have shown that our method outperforms state-of-the-art methods by remarkable margins of 7.2%, 3.7%, and 9.4% in terms of 3D mean per joint position error (MPJPE) on the Human3.6M, CMU Mocap, and 3DPW datasets, respectively. We also demonstrate that our method is more robust under data missing cases and noisy data cases.
- Cuda 11.1
- Python 3.7
- Pytorch 1.8.0
Download Human3.6M dataset from its website and put the files into "data" with the dataset folder named "h3.6m".
Download CMUMocap dataset from its website and put the files into "data" with the dataset folder named "CMUMocap".
Download 3DPW dataset from its website and put the files into "data" with the dataset folder named "3DPW".
After the prepartion work, the data folder should have the following structure:
/data
├── h3.6m
├── dataset
├── S1
├── S5
├── ...
├── CMUMocap
├── test
├── train
├── 3DPW
├── sequenceFiles
To train a short-term motion prediction model, run
CUDA_VISIBLE_DEVICES={GPU_ID} python main_h36m.py --task short
To train a long-term motion prediction model, run
CUDA_VISIBLE_DEVICES={GPU_ID} python main_h36m.py --task long
The trained model will be saved in ckpt/ .
To train a short-term motion prediction model, run
CUDA_VISIBLE_DEVICES={GPU_ID} python main_cmu.py --task short
To train a long-term motion prediction model, run
CUDA_VISIBLE_DEVICES={GPU_ID} python main_cmu.py --task long
The trained model will be saved in ckpt/ .
To train a short-term motion prediction model, run
CUDA_VISIBLE_DEVICES={GPU_ID} python main_3dpw.py --task short
To train a long-term motion prediction model, run
CUDA_VISIBLE_DEVICES={GPU_ID} python main_3dpw.py --task long
The trained model will be saved in ckpt/ .
Take our trained model as an example, to evaluate a short-term motion prediction model, run
CUDA_VISIBLE_DEVICES={GPU_ID} python test_h36m.py --task short --model_save_name pretrain_h36m_ckpt
To evaluate a long-term motion prediction model, run
CUDA_VISIBLE_DEVICES={GPU_ID} python test_h36m.py --task long --model_save_name pretrain_h36m_ckpt_long
Take our trained model as an example, to evaluate a short-term motion prediction model, run
CUDA_VISIBLE_DEVICES={GPU_ID} python test_cmu.py --task short --model_save_name pretrain_cmu_ckpt
To evaluate a long-term motion prediction model, run
CUDA_VISIBLE_DEVICES={GPU_ID} python test_cmu.py --task long --model_save_name pretrain_cmu_ckpt_long
Take our trained model as an example, to evaluate a short-term motion prediction model, run
CUDA_VISIBLE_DEVICES={GPU_ID} python test_3dpw.py --task short --model_save_name pretrain_3dpw_ckpt
To evaluate a long-term motion prediction model, run
CUDA_VISIBLE_DEVICES={GPU_ID} python test_3dpw.py --task long --model_save_name pretrain_3dpw_ckpt_long
We thank for the part of the code provided by LearnTrajDep and SPGSN.
If you find our work useful in your research, please cite our paper:
@inproceedings{xu2023auxiliary,
title={Auxiliary Tasks Benefit 3D Skeleton-based Human Motion Prediction},
author={Xu, Chenxin and Tan, Robby T and Tan, Yuhong and Chen, Siheng and Wang, Xinchao and Wang, Yanfeng},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={9509--9520},
year={2023}
}