/efficientpose

EfficientPose: Body Segmentation for TFJS and NodeJS

Primary LanguageJavaScript

EfficientPose: Body Segmentation for TFJS and NodeJS

Models included in /model-tfjs-graph-* were converted to TFJS Graph model format from the original repository
Models descriptors have been additionally parsed for readability


Implementation

Actual model parsing implementation in efficientpose.js does not follow original
and is implemented using native TFJS ops and optimized for JavaScript execution

Function processResults() takes model.execute output and returns array of 16 keypoints:

  • id
  • score: score as number
  • label: annotated body part as string
  • xRaw: x coordinate normalized to 0..1
  • yRaw: y coordinate normalized to 0..1
  • x: x coordinate normalized to input image size
  • y: y coordinate normalized to input image size



Example

Example Image

Test

node efficientpose.js body.jpg
2021-03-24 15:41:37 INFO:  efficientpose version 0.0.1
2021-03-24 15:41:37 INFO:  User: vlado Platform: linux Arch: x64 Node: v15.12.0
2021-03-24 15:41:37 INFO:  Loaded model { modelPath: 'file://models/iv/efficientpose.json', minScore: 0.2 } tensors: 955 bytes: 25643252
2021-03-24 15:41:37 INFO:  Model Signature {
  inputs: { input_res1: { name: 'input_res1', dtype: 'DT_FLOAT', tensorShape: { dim: [ { size: '1' }, { size: '600' }, { size: '600' }, { size: '3' } } } },
  outputs: { 'upscaled_confs/BiasAdd:0': { name: 'upscaled_confs/BiasAdd:0', dtype: 'DT_FLOAT', tensorShape: { dim: [ { size: '1' }, { size: '-1' }, { size: '-1' }, { size: '16' } } } }
}
2021-03-24 15:41:37 INFO:  Loaded image: body.jpg inputShape: [ 1024, 1024, 3 ] modelShape: [ 1, 600, 600, 3 ] decoded size: 3145728
2021-03-24 15:41:39 DATA:  Results: [
  { id: 0, score: 0.8234584331512451, label: 'head', xRaw: 0.4033333333333333, yRaw: 0.051666666666666666, x: 413, y: 53 },
  { id: 1, score: 0.8789138197898865, label: 'neck', xRaw: 0.4533333333333333, yRaw: 0.18166666666666667, x: 464, y: 186 },
  { id: 2, score: 0.8490188717842102, label: 'rightShoulder', xRaw: 0.395, yRaw: 0.205, x: 404, y: 210 },
  { id: 3, score: 0.8640593886375427, label: 'rightElbow', xRaw: 0.40166666666666667, yRaw: 0.3333333333333333, x: 411, y: 341 },
  { id: 4, score: 0.8743583559989929, label: 'rightWrist', xRaw: 0.4066666666666667, yRaw: 0.45666666666666667, x: 416, y: 468 },
  { id: 5, score: 0.8736196756362915, label: 'chest', xRaw: 0.46166666666666667, yRaw: 0.21166666666666667, x: 473, y: 217 },
  { id: 6, score: 0.8904648423194885, label: 'leftShoulder', xRaw: 0.5283333333333333, yRaw: 0.215, x: 541, y: 220 },
  { id: 7, score: 0.9026476144790649, label: 'leftElbow', xRaw: 0.525, yRaw: 0.3616666666666667, x: 538, y: 370 },
  { id: 8, score: 0.7956844568252563, label: 'leftWrist', xRaw: 0.47333333333333333, yRaw: 0.49166666666666664, x: 485, y: 503 },
  { id: 9, score: 0.8972961902618408, label: 'pelvis', xRaw: 0.5066666666666667, yRaw: 0.45666666666666667, x: 519, y: 468 },
  { id: 10, score: 0.807637631893158, label: 'rightHip', xRaw: 0.4666666666666667, yRaw: 0.45666666666666667, x: 478, y: 468 },
  { id: 11, score: 0.8232259750366211, label: 'rightKnee', xRaw: 0.47833333333333333, yRaw: 0.63, x: 490, y: 645 },
  { id: 12, score: 0.9226986765861511, label: 'rightAnkle', xRaw: 0.43833333333333335, yRaw: 0.79, x: 449, y: 809 },
  { id: 13, score: 0.7791210412979126, label: 'leftHip', xRaw: 0.545, yRaw: 0.4533333333333333, x: 558, y: 464 },
  { id: 14, score: 0.8537712097167969, label: 'leftKnee', xRaw: 0.5883333333333334, yRaw: 0.65, x: 602, y: 666 },
  { id: 15, score: 0.8724350333213806, label: 'leftAnkle', xRaw: 0.6016666666666667, yRaw: 0.8433333333333334, x: 616, y: 864 },
]
2021-03-24 15:41:39 STATE:  Created output image: outputs/body.jpg size: [ 1024, 1024 ]



Conversion Notes

Original: https://github.com/daniegr/EfficientPose

Requirements

Edit requirements.txt to remove specific version pinning and install required packages:

sudo apt install libmediainfo-dev
pip install -r requirements.txt
pip install tensorflowjs

Test

Edit track.py to fix tensor names:

    # TensorFlow
    elif framework in ['tensorflow', 'tf']:
        output_tensor = model.graph.get_tensor_by_name('upscaled_confs/BiasAdd:0')
        if lite:
            batch_outputs = model.run(output_tensor, {'input_1_0:0': batch})            
        else:
            batch_outputs = model.run(output_tensor, {'input_res1:0': batch})

Run test:

python track.py --path=body.jpg --model=II_Lite --framework=tensorflow --visualize --store

Convert

From TensorFlow Frozen model to TFJS Graph model:

tensorflowjs_converter \
--input_format tf_frozen_model \
--output_format tfjs_graph_model \
--strip_debug_ops=* \
--weight_shard_size_bytes=16777216 \
--output_node_names='upscaled_confs/BiasAdd:0' \
tensorflow/EfficientPoseII_LITE.pb \
tfjs/ii-lite

After conversion, lets add correct model signature in model.json

  "signature": {
      "inputs": { "input_1_0": { "name": "input_1_0", "dtype": "DT_FLOAT", "tensorShape":{"dim":[{"size":"1"},{"size":"368"},{"size":"368"},{"size":"3"}]} } },
      "outputs": { "upscaled_confs/BiasAdd:0": { "name": "upscaled_confs/BiasAdd:0", "dtype": "DT_FLOAT", "tensorShape":{"dim":[{"size":"1"},{"size":"-1"},{"size":"-1"},{"size":"16"}]} } }
  },