/handsfree-browser

A browser extension for using and developing the web handsfree, powered by Handsfree.js

Primary LanguageJavaScriptMozilla Public License 2.0MPL-2.0

💥💥💥 This project has been archived and being merged into the Handsfree.js repo to create a tighter iteration cycle 💥💥💥







The Handsfree Browser Extension

🚧🐞 This project is useable but not officially released and still buggy 🐞🚧

This Handsfree.js powered browser extension is a designed to help you browse the web handsfree through face and/or hand gestures. The goal is to develop a "Userscript Manager" like Tampermonkey, but for handsfree-ifying web pages, games, apps, WebXR and really any other type of content found the web.

This browser extension also includes a handsfree fork of the Mozilla WebXR Emulator Extension Dev Tools, primarily to assist in handsfree WebXR development.

In short, this extension will help you use your entire browser, desktop, or connected drones, robots, and more in customizable and interesting new ways 🖐👀🖐







Handsfree Browsing

By default, each page will get a "Face Pointer" or a set of "Palm Pointers" for you to browse pages handsfree.

A person controlling a virtual mouse pointer by tilting their head around A person scrolling a page by pinching their index and thumb together and raising or lowering their pinched hand

However, in addition to the pointers you can add custom handsfree interactions. For example, here's a demonstration of handsfree-ifying different things:

Creating generative art with hand gestures A person pinching the air to "pinch" a blob. Moving a pinched blob causes it to sing in a different pitch







Handsfree WebXR DevTools

This project is a started as a fork of the Mozilla WebXR Emulator Extension. Currently the DevTools helps you emulate WebXR headsets with basic controller support.







Development Guide

Folder structure

Each of the files are located in their respective context folders in /src/. Handsfree.js specific scripts are named handsfree.js and the Mozilla WebXR specific ones are labeled webxr.js. Other than organizing the files the WebXR code is mostly untouched.

How it works

  • When you first install the extension, /src/background/handsfree.js checks if you've approved the webcam. If not, then it'll open the options page in src/options/stream-capture.html
  • The popup panel has a "Start/Stop Webcam" button that communicates with the background script to start the webcam: /src/popup/index.html
  • The background page is where the models are stored and run. This keeps everything isolated and only asks for webcam permission once (vs on every domain): /src/background/handsfree.js
  • The background page also uses the "Picture in Picture" API to "pop the webcam" out of the browser. It renders the webcam feed and debug canvases into a single canvas, and uses that as the srcObject to a separate video element which is the PiP'ed

How to install

  • Chrome: Install as an unpacked chrome extension. Visit chrome://extensions and enable Developer Mode on the top right, then click Load unpacked and select this project's root folder

  • Firefox: Coming soon, I switched computers and didn't realize I continued developing on Chrome 😆






Acknowledgements