Realtime Multi-Person Pose Estimation
By Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh.
Introduction
Code repo for winning 2016 MSCOCO Keypoints Challenge, ECCV Best Demo Award.
Watch our [video result] (https://www.youtube.com/watch?v=pW6nZXeWlGM&t=77s) on funny Youtube videos.
We present a bottom-up approach for multi-person pose estimation, without using any person detector. For more details, refer to our Arxiv paper and presentation slides at ILSVRC and COCO workshop 2016.
This project is licensed under the terms of the GPL v3 license .
Contact: Zhe Cao Email: zhecao@cmu.edu
Results
Contents
Testing
C++ (realtime version)
- Use our modified caffe: caffe_rtpose. Follow the instruction on that repo.
- Three input options: images, video, webcam
Matlab (slower)
- Compatible with general Caffe. Compile matcaffe.
- Run
cd testing; get_model.sh
to retreive our latest MSCOCO model from our web server. - Change the caffepath in the
config.m
and rundemo.m
for an example usage.
Python
- iPython Notebook documentation will be released soon!
Training
Network Architecture
Usage
- Use our modified caffe: caffe_train. Compile pycaffe. It will be merged with caffe_rtpose (for testing) soon.
- Run
cd training; python setLayers.py --exp 1
to generate the prototxt and shell file for training. - Download VGG-19 model, we use it to initialize the first 10 layers for training.
- Download our generated LMDB for the COCO dataset (189GB file):
get_lmdb.sh
- Code for generating the LMDB file will be released soon!
Related repository
CVPR'16, Convolutional Pose Machines
Citation
Please cite the paper in your publications if it helps your research:
@article{cao2016realtime,
title={Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields},
author={Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh},
journal={arXiv preprint arXiv:1611.08050},
year={2016}
}
@inproceedings{wei2016cpm,
author = {Shih-En Wei and Varun Ramakrishna and Takeo Kanade and Yaser Sheikh},
booktitle = {CVPR},
title = {Convolutional pose machines},
year = {2016}
}