/handsfree

Build handsfree User Experiences and add face, hand, and/or pose tracking to your projects in a snap ✨👌

Primary LanguageJavaScriptApache License 2.0Apache-2.0

handsfree.js.org

Build handsfree User Experiences and add face, hand, and pose tracking to your projects in a snap 👌✨

Powered by:

       







Explore the interactive docs at: Handsfree.js.org

Or try it right away with the serverless boilerplates in /boilerplate/!







Quickstart

Installing from CDN

This option is great if you don't have or need a server, or if you're developing on a site like CodePen. You can also just download this repo and open up /boilerplates/cdn.html in your browser. Since the models are loaded on CDN you won't need to do any further setup to get started!

<head>
  <!-- Include Handsfree.js -->
  <link rel="stylesheet" href="https://unpkg.com/handsfree@8.2.0/build/lib/assets/handsfree.css" />
  <script src="https://unpkg.com/handsfree@8.2.0/build/lib/handsfree.js"></script>
</head>

<body>
  <!-- Your code must be inside body as it applies classes to it -->
  <script>
    // Instantiate and start Handsfree
    const handsfree = new Handsfree({hands: true})
    handsfree.start()

    // Create a plugin named "logger" to show data on every frame
    handsfree.use('logger', data => {
      console.log(data)
    })
  </script>
</body>

Installing from NPM

# From your projects root
npm i handsfree
// Inside your app
import Handsfree from 'handsfree'

const handsfree = new Handsfree({hands: true})
handsfree.enablePlugins('browsing')
handsfree.start()

Hosting the models yourself

The above will load models, some over 10Mb, from the Unpkg CDN. If you'd rather host these yourself (for example, to use offline) then you can eject the models from the npm package into your project's public folder:

# Move the models into your project's public directory
# - change PUBLIC below to where you keep your project's assets

# ON WINDOWS
xcopy /e node_modules\handsfree\build\lib PUBLIC
# EVERYWHERE ELSE
cp -r node_modules/handsfree/build/lib/* PUBLIC
import Handsfree from 'handsfree'

const handsfree = new Handsfree({
  hands: true,
  // Set this to your where you moved the models into
  assetsPath: '/PUBLIC/assets',
})
handsfree.enablePlugins('browsing')
handsfree.start()






Example Workflow

The following aims to give you a quick overview of how things work. The key takeaway is that everything is centered around hooks/plugins, which are basically named callbacks which are run on every frame and can be toggled on and off.

Quickstart Workflow

The following workflow demonstrates how to use all features of Handsfree.js. Check out the Guides and References to dive deeper, and feel free to post on the Google Groups or Discord if you get stuck!

// Let's enable face tracking with the default Face Pointer
const handsfree = new Handsfree({weboji: true})
handsfree.enablePlugins('browsing')

// Now let's start things up
handsfree.start()

// Let's create a plugin called "logger"
// - Plugins run on every frame and is how you "plug in" to the main loop
// - "this" context is the plugin itself. In this case, handsfree.plugin.logger
handsfree.use('logger', data => {
  console.log(data.weboji.morphs, data.weboji.rotation, data.weboji.pointer, data, this)
})

// Let's switch to hand tracking now. To demonstrate that you can do this live,
// let's create a plugin that switches to hand tracking when both eyebrows go up
handsfree.use('handTrackingSwitcher', {weboji} => {
  if (weboji.state.browsUp) {
    // Disable this plugin
    // Same as handsfree.plugin.handTrackingSwitcher.disable()
    this.disable()

    // Turn off face tracking and enable hand tracking
    handsfree.update({
      weboji: false,
      hands: true
    })
  }
})

// You can enable and disable any combination of models and plugins
handsfree.update({
  // Disable weboji which is currently running
  weboji: false,
  // Start the holistic model
  holistic: true,

  // This is also how you configure (or pre-configure) a bunch of plugins at once
  plugin: {
    fingerPointer: {enabled: false},
    faceScroll: {
      vertScroll: {
        scrollSpeed: 0.01
      }
    }
  }
})

// Disable all plugins
handsfree.disablePlugins()
// Enable only the plugins for making music (not actually implemented yet)
handsfree.enablePlugins('music')

// Overwrite our logger to display the original model APIs
handsfree.plugin.logger.onFrame = (data) => {
  console.log(handsfree.model.holistic?.api, handsfree.model.weboji?.api, handsfree.model.pose?.api)
}






Local Development

Running Handsfree.js.org locally

  • Install NodeJS
  • Download this repository: git clone https://github.com/handsfreejs/handsfree
  • Install dependencies by running npm i from the project root
  • Start development on localhost:8080 by running npm start

Command line scripts

# Start local development on localhost:8080
npm start 

# Build into /dist/lib/ and /dist/docs/
npm run build

# Build only /dist/docs/
npm run build:docs

# Build only /dist/lib/
npm run build:lib

Dev Notes

  • See vuepress-component-font-awesome for adding fonts. Remember to run npm run fa:build when adding new fonts so that they are copied over into the docs/.vuepress/components/FA folder
  • You may occasionally need to restart server when adding new files to the /docs, this is true when changing /docs/.vuepress.config.js as well






Explore the interactive docs at: Handsfree.js.org

Or try it right away with the serverless boilerplates in /boilerplate/!







Examples

Face Tracking Examples

Face Pointers

Motion Parallax Display

Puppeteering Industrial Robots

Playing desktop games with face clicks

Hand Tracking Examples

Hand Pointers

Use with Three.js

Playing desktop games with pinch clicks

Laser pointers but with your finger

Pose Estimation Examples

Flappy Pose - Flappy Bird but where you have to flap your arms






License & Attributions

License: Apache 2.0

The Handsfree.js core is available for free and commercial use under Apache 2.0. Each of the models are also available for free and commercial use under Apache 2.0:

Attributions

I'd like to also thank the following people and projects:







Thanks to the Handsfree.js.org GitHub Sponsors

Starting a large project and movement is so much fun but it's also a bit daunting 😅 I'd like to thank the following for supporting this project:

I'd also like to thank:







Special Thanks