/openpose_training

Code repo for realtime multi-person pose estimation in CVPR'17 (Oral)

Primary LanguageJupyter NotebookOtherNOASSERTION

Realtime Multi-Person Pose Estimation

By Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh.

Introduction

Code repo for winning 2016 MSCOCO Keypoints Challenge, 2016 ECCV Best Demo Award, and 2017 CVPR Oral paper.

Watch our video result in YouTube or our website.

We present a bottom-up approach for multi-person pose estimation, without using any person detector. For more details, refer to our CVPR'17 paper or our presentation slides at ILSVRC and COCO workshop 2016.

This project is licensed under the terms of the license.

Contact: Zhe Cao Email: zhecao@cmu.edu

Results

Contents

  1. Testing
  2. Training
  3. Citation

Testing

C++ (realtime version, for demo purpose)

  • Use our modified caffe: caffe_rtpose. Follow the instruction on that repo.
  • In May 2017, we released an updated library OpenPose.
  • Three input options: images, video, webcam.

Matlab (slower, for COCO evaluation)

  • Compatible with general Caffe. Compile matcaffe.
  • Run cd testing; get_model.sh to retreive our latest MSCOCO model from our web server.
  • Change the caffepath in the config.m and run demo.m for an example usage.

Python

  • cd testing/python.
  • ipython notebook.
  • Open demo.ipynb and execute the code.

Training

Network Architecture

Teaser?

Training Steps

  • a) Generate data for training

    • Option a)
      • Run cd training; bash a0_getData.sh to obtain the COCO images in dataset/COCO/images/, keypoints annotations in dataset/COCO/annotations/ and COCO official toolbox in dataset/COCO/coco/.
      • Run a1_jsonToMat.m in Matlab to convert the annotation format from json to mat in dataset/COCO/mat/.
      • Run a2_matToMasks.m in Matlab to obatin the mask images for unlabeled person. You can use 'parfor' in Matlab to speed up the code.
      • Run a3_matToRefinedJson.m to generate a json file in dataset/COCO/json/ folder. The json files contain raw informations needed for training.
      • Run python a4_genLMDB.py to generate your LMDB.
    • Option b)
      • Alternatively, you could simply download our prepared LMDB for the COCO dataset (189GB file) by: bash get_lmdb.sh.

  • b) Custom Caffe: Download and compile our modified Caffe

    • Download link: caffe_train.
    • Compile it by running: make all -j{num_cores} && make pycaffe -j{num_cores}.
    • Note: It will be merged with caffe_rtpose (for testing) soon.

  • c) Generate Caffe ProtoTxt and shell file for training

    • Run python c_setLayers.py --exp 1.

  • d) Pre-trained model (VGG-19): Get pre-trained model to initialize the first 10 layers of the network for training

  • e) Train

    • Run bash train_pose.sh 0,1,2,3 (generated by setLayers.py) to start the training with 4 GPUs (0-3).

Related Repositories

Citation

Please cite the paper in your publications if it helps your research:

@inproceedings{cao2017realtime,
  author = {Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh},
  booktitle = {CVPR},
  title = {Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields},
  year = {2017}
  }

@inproceedings{wei2016cpm,
  author = {Shih-En Wei and Varun Ramakrishna and Takeo Kanade and Yaser Sheikh},
  booktitle = {CVPR},
  title = {Convolutional pose machines},
  year = {2016}
  }