/posenet2scratch

Pose Detection on Scratch 3 using PoseNet

Primary LanguageJavaScriptBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

PoseNet2Scratch

Read this in other languages: English, 日本語

PoseNet2Scratch can detect human pose and add special Scratch blocks that can get x and y position of each part of the body.

It uses PoseNet, a machine learning model , for Real-time Human Pose Estimation. PoseNet advantage is that it can detect human poses quickly and accurately with a normal WebCam without using a special device such as Kinect.

ml5.js, a JavaScript library for machine learning, is used to connect it with Scratch 3.

Try the customized Scratch 3 that has PoseNet2Scratch as an extension.

Licence

  • PoseNet2Scratch is under BSD 3-Clause License, open source and freely available to anyone. You can use it at your classes, workshops. Commercial usage is also accepted. If you or your students created something cool using PoseNet2Scratch, please share it on SNS using hashtag #posenet2scratch or let me know to any of these contacts.

Contacts

  • Twitter

Requirements

  • OS
    • Windows 8
    • Windows 10
    • MacOS
  • Browser
    • Chrome

PoseNet2Scratch sometimes does not work because of some Chrome extensions. Plese switch to Guest Mode in such cases.

How to use

  1. Open https://champierre.github.io/scratch3/ on a browser(Google Chrome is recommended). PoseNet2Scratch sometimes does not work because of some Chrome extensions. Plese switch to Guest Mode in such cases.

  2. Click "Add Extension" button(Folder with "+" mark icon).

  1. Select "PoseNet2Scratch" extension.

  1. You can use PoseNet2Scratch blocks.

  1. Select "Video Sensing" extension, if you want to see the image captured by webcam on the stage screen. This is not mandatory.

  1. Add "Glasses" sprite.

  1. Make a code like below.

  1. You will have the glasses shown at the center of your face(x position of the nose) and at the hight of your eyes(y position of the left eye).

With PoseNet2Scratch extension blocks, you can get x and y position of the following body parts:

  • nose
  • left eye
  • right eye
  • left ear
  • right ear
  • left shoulder
  • right shoulder
  • left elbow
  • right elbow
  • left wrist
  • right wrist
  • left hip
  • right hip
  • left knee
  • right knee
  • left ankle
  • right ankle