PeriniM/HandsEase

HandsEase Open Project

Open source template for hand detection webapps. You can build drawing apps, manipulate 3D objects on the web and so on.
All you need is a webcam!

Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgments

About The Project

Open source template for hand detection webapps. You can build drawing apps, manipulate 3D objects on the web and so on.

(back to top)

Built With

(back to top)

Getting Started

To get a local copy up and running follow these simple example steps.

Prerequisites

• Install Node.js from the main website here

Optional:

• Run the Docker container using the Visual Studio Code Dev Containers extension

Installation

1. Clone the repo

   git clone https://github.com/PeriniM/HandsEase.git

2. Install NPM packages

   npm install

3. To run it in a local web server

   npm run start

4. If you just want to build it

   npm build

(back to top)

Usage

Drawing.App.mp4

For more examples, please refer to the Documentation

(back to top)

Roadmap

• Folders organization: https://mapstore2.readthedocs.io/en/new_merge_doc/developer-guide/project-structure/

DETECTION MODEL

• Use tensorflowjs model
  • https://github.com/tensorflow/tfjs-models/tree/master/hand-pose-detection
  • https://machinelearningknowledge.ai/tensorflow-js-hand-gesture-recognition-and-tracking-using-handpose-model/
• Move the computation to a parallel "thread" using Web Workers (working only on tfjs models)
  • https://stackoverflow.com/questions/64249599/how-to-run-handpose-tfjs-model-in-web-worker
  • https://hacks.mozilla.org/2011/12/faster-canvas-pixel-manipulation-with-typed-arrays/
• Implement Linear Kalman Filter
  • https://ethz.ch/content/dam/ethz/special-interest/baug/igp/photogrammetry-remote-sensing-dam/documents/pdf/Student_Theses/BA_Laumer-Gumgumcu_final.pdf
  • https://nbviewer.org/github/rlabbe/Kalman-and-Bayesian-Filters-in-Python/blob/master/00-Preface.ipynb
• Implement gesture detection model

GRAPHICS

• Port the app in threejs
• Add 3D Gizmo
• Add grid floor
• Add a mockup for the rig of the hands (like transparent hands 3d model)

INTERACTION

• "OK" gesture should pick things and also draw
• Implement rotation and pan with the other hand
• Double "OK" gesture allow the user to rotate and scale objects

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

Contact

Marco Perini - @linkedin

Project Link: HandsEase Repo

(back to top)

Acknowledgments

(back to top)