ATMx 🏧

An Interactive (Proof of Concept) Web Demo showcasing Touchless Interactions using the MediaPipe Machine Learning Library ✨

Touchless gesture-based systems hold the potential to reshape public spaces, workplaces, and industries as it's seamless and convenient nature aligns with the post-pandemic landscape. This innovative webapp harnesses the power of the MediaPipe Handlandmarker Task, featuring an augmented transaction panel that enables users to interact effortlessly through intuitive hand gestures, detected from the input video feed. Notably, user data collected from the video feed is promptly deleted after inference, ensuring GDPR compliance as all computations occur client-side. Users can seamlessly perform essential operations via the on-screen interactive panel, utilizing straightforward custom gestures to navigate the checkout process without the need for physical touch, and it can even run offline! I was able to pioneer a novel method by leveraging custom logic to detect hand gestures directly via the HandLandmarker task, eliminating the reliance on heavyweight dependencies like the GestureRecognizer Task. This optimization not only streamlined the project but also contributed to enhancing the core MediaPipe framework's efficiency. The process involves segmenting the video stream to identify hands, localizing the palm area, and subsequently tracking specific hand keypoints for precise gesture recognition. This comprehensive approach significantly enhances performance, seamlessly integrating with the default Handlandmarker API. This project is a symbol of my commitment to embracing the future, harnessing innovation, and crafting a more hygienic, efficient, and enchanting way to connect with technology!

This project aims to test and demonstrate the capabilities of the new MediaPipe Hand Landmarker task from MediaPipe Solutions, which outputs 21 hand landmarks. The task provides precise and accurate hand landmark detection, generating 21 key points on the hand. These landmarks are utilized in this interactive web app which enables users to perform contactless interactions with the interface using simple human gestures. Best experienced in well-lit environments. Ideal on larger screens.

Shortened Version

In a rapidly evolving technological landscape, the aftermath of the COVID-19 pandemic has amplified concerns regarding hygiene and touch-based interactions. With 80% of individuals deeming public touchscreens unhygienic, there is a compelling need for innovative solutions. Enter touchless gesture-based systems, poised to reshape industries and public spaces. Seamlessly aligning with the post-pandemic era, this technology offers intuitive and convenient interactions. From ATMs and airports to healthcare and retail, touchless interactions are on the brink of becoming ubiquitous. This project directly addresses these changing expectations by harnessing the power of the MediaPipe Hand Landmarker task from MediaPipe Solutions. By precisely detecting 21 key hand landmarks, this technology powers an interactive web application enabling users to effortlessly engage with interfaces through contactless gestures. Designed for optimal performance in well-lit environments and on larger screens, this project embodies the future of safer, more advanced interactions.

Prerequisites

This project requires Node.js to be installed on your local machine.

⚠️ Webcam required for hand detection and gesture recognition. Please ensure your device has a functioning webcam.

Installation

Clone the repository on your local machine:

git clone https://github.com/Neilblaze/ATMx.git

Navigate into the project directory:
```
cd ATMx
```
Install the necessary dependencies:
```
npm install
```
Start the development server:
```
npm start
```
Open the project in your browser at http://localhost:3000 to view your project.

🚀 View a live demo in your browser here.

Built With

This project was created using:

Case Study

This project is part of my contribution as a mentee in Google Summer of Code 2023, working under the TensorFlow org, & is a symbol of my commitment to embracing the future, harnessing innovation, and crafting a more hygienic, efficient, and enchanting way to connect with technology!

I wrote a comprehensive article on this project, which you can read here.

ⓘ This is intended to assist other developers in utilizing the MediaPipe library and implementing similar touchless interaction features in their projects.

Architecture Overview 🔻

ⓘ If you want to delve deep into the specs of the model, feel free to explore the official docs, which can be found here. You can access the official model card for MediaPipe Hands (Lite/Full) here. It provides detailed information about the model.

References

[1] MediaPipe Hands Official Paper: (LINK🔗)

[2] Applying Hand Gesture Recognition for User Guide Application Using MediaPipe (Paper): (LINK🔗)

[3] MediaPipe Solutions API Docs: (LINK🔗)

Update (v0.1.1)

Precaching is enabled! Hence, the app will work offline after the first load. 🎉

Contact

Pratyay Banerjee — dev@neilblaze.live

License

Distributed under the Apache License 2.0. See LICENSE for more information.

atlury/ATMx