/frankmocap

A Strong and Easy-to-use Single View 3D Hand+Body Pose Estimator

Primary LanguagePythonOtherNOASSERTION

FrankMocap: A Strong and Easy-to-use Single View 3D Hand+Body Pose Estimator

FrankMocap pursues an easy-to-use single view 3D motion capture system developed by Facebook AI Research (FAIR). FrankMocap provides state-of-the-art 3D pose estimation outputs for body, hand, and body+hands in a single system. The core objective of FrankMocap is to democratize the 3D human pose estimation technology, enabling anyone (researchers, engineers, developers, artists, and others) can easily obtain 3D motion capture outputs from videos and images.

News:

  • [2020/10/09] We have improved openGL rendering speed. It's about 40% faster. (e.g., body module: 6fps -> 11fps)

Key Features

  • Body Motion Capture:

  • Hand Motion Capture

  • Egocentric Hand Motion Capture

  • Whole body Motion Capture (body + hands)

Installation

A Quick Start

  • Run body motion capture

    # using a machine with a monitor to show output on screen
python -m demo.demo_bodymocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output

# screenless mode (e.g., a remote server)
xvfb-run -a python -m demo.demo_bodymocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output

  • Run hand motion capture

    # using a machine with a monitor to show outputs on screen
python -m demo.demo_handmocap --input_path ./sample_data/han_hand_short.mp4 --out_dir ./mocap_output

# screenless mode  (e.g., a remote server)
xvfb-run -a python -m demo.demo_handmocap --input_path ./sample_data/han_hand_short.mp4 --out_dir ./mocap_output

  • Run whole body motion capture

    # using a machine with a monitor to show outputs on screen
python -m demo.demo_frankmocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output

# screenless mode  (e.g., a remote server)
xvfb-run -a python -m demo.demo_frankmocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output

  • Note:

    • Above commands use openGL by default. If it does not work, you may try alternative renderers (pytorch3d or openDR).
    • See the readme of each module for details

Body Motion Capture Module

Hand Motion Capture Module

Whole Body Motion Capture Module (Body + Hand)

License

References

  • FrankMocap is based on the following research outputs:
@article{rong2020frankmocap,
  title={FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration},
  author={Rong, Yu and Shiratori, Takaaki and Joo, Hanbyul},
  journal={arXiv preprint arXiv:2008.08324},
  year={2020}
}

@article{joo2020eft,
  title={Exemplar Fine-Tuning for 3D Human Pose Fitting Towards In-the-Wild 3D Human Pose Estimation},
  author={Joo, Hanbyul and Neverova, Natalia and Vedaldi, Andrea},
  journal={arXiv preprint arXiv:2004.03686},
  year={2020}
}