hand-gestures-smart-home

Overview

Interact with smart home via hand gestures.

Installation

Clone repository

git clone git@github.com:chaichana-t/hand-gestures-smart-home.git
cd hand-gestures-smart-home

Install dependencies (recommend python version >= 3.9)

pip install -r requirements.txt

You can config constant like time interval or bulb ip in config.py

How to use

Run app

python main.py

Example

time_interval = 2
bulb_ips = ['192.168.1.103','192.168.1.102']

Supporting

Supporting smart home interactions
- Turn on / off
- Increase brightness
- Decrease brightness
- Coloring
  - Temperature
  - Randomly pick
  - Disco

Information

Problem Statement

Voice is being used as an input for interaction in smart home devices. However, some people aren’t be able to use their voice like everyone else. For example, Deaf people. We recognize the difficulty and want to ensure that everyone has access to smart home technology. As a result, we'll be utilizing computer vision to address this issue.

Technical Challenges

We can't predict if users would gesture only the actions that are presented in real-world settings. As a result, we've utilized certain strategies to overcome, which we'll discuss in the upcoming session.

Related Works

From our researches we found two possible approaches for the work

Real-time Hand Gesture Recognition: link
- Detect hand using Google MediaPipe.
- Recognize hand gesture using hand landmarks using Neural Networks.
Hand Detection using Object Detection
- Detect hand with its action using object detection approach.

Method and Results

AI
- Hand Detection
- Hand Gesture Recognition
APP
- Motion Interval
- Yeelight connection

AI

Use two stages model

Hand Detection
Hand Gesture Recognition

Hand Detection

Google MediaPipe (ref. Github)

Hand Gesture Recognition

Data
- Collected by ourselves
- Data distribution

Model

Fully Connected Neural Network

Input: Coordinate x, y of each key points

Normalized by position and size

def normalize(landmarks):
    base_x, base_y = landmarks[0][0], landmarks[0][1]
    landmarks = [[landmark[0] - base_x, landmark[1] - base_y]
                  for landmark in landmarks]
    max_value = max([max([abs(x) for x in landmark])
                  for landmark in landmarks])
    landmarks = [[landmark[0] / max_value, landmark[1] / max_value]
                  for landmark in landmarks]
    return landmarks

Structure: [[x0, y0], [x1, y1], ..., [x20, y20]]

Output: Action

Model Architecture

class HandGestureModel(nn.Module):

    def __init__(self):
        super(HandGestureModel, self).__init__()

        self.classifier = nn.Sequential(
            nn.Linear(in_features=42, out_features=64),
            nn.ReLU(),
            nn.Dropout(p=0.2),
            nn.Linear(in_features=64, out_features=128),
            nn.ReLU(),
            nn.Dropout(p=0.2),
            nn.Linear(in_features=128, out_features=512),
            nn.ReLU(),
            nn.Dropout(p=0.2),
            nn.Linear(in_features=512, out_features=64),
            nn.ReLU(),
            nn.Dropout(p=0.2),
            nn.Linear(in_features=64, out_features=32),
            nn.ReLU(),
            nn.Linear(in_features=32, out_features=6),
        )

        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        x = torch.flatten(x, start_dim=1)
        x = self.classifier(x)
        return self.sigmoid(x)

Our Contribute (from Technical Challenges)
- This problem is Multiclass Single-label Classification
  - Typical Softmax and CrossEntropyLoss were applied.
    - Model can recognize every action with high confidence.
    - What about edge cases if we don’t have in dataset?
  - We applied Sigmoid and BCELoss for idealy minimizing all action's probabilities of edge cases to zero.
    - Train it like Multiclass Multi-label Classification (but each label has an 1’s in one-hot encoding)
    - Inference using max confidence and thresholding 0.999
    - However, this does not perfectly work.

Result

Model Training

Model Evaluation

Classification Report

              precision    recall  f1-score   support

           0     1.0000    1.0000    1.0000       613
           1     1.0000    1.0000    1.0000       556
           2     1.0000    1.0000    1.0000       143
           3     1.0000    1.0000    1.0000       618
           4     1.0000    1.0000    1.0000       550
           5     1.0000    1.0000    1.0000       458

    accuracy                         1.0000      2938
   macro avg     1.0000    1.0000    1.0000      2938
weighted avg     1.0000    1.0000    1.0000      2938

Confusion Matrix

APP

Motion Interval
- We use time as interval in order to request Yeelight API which currently is 2 seconds.
Yeelight connection
- We use YeeLight python library to control our smart light bulb.
- Our actions that could be used by the model
  - Turn on / off (toggle)
  - Increase brightness
  - Decrease brightness
  - Coloring
    - Temp (toggle)
    - Randomly pick
    - Disco
Gesture Action

Turn on / off (toggle) Increase brightness Decrease brightness

Temp (toggle) Randomly pick Disco

Limitations and Future Work

AI

Sufficient lighting is required
Improve model performance such that actions that aren't supplied aren't recognized
Improve model more generalized to recognize all possible hand gestures

APP

Improve UX/UI
Port it onto smaller device
Expand it to more smart devices (Not only Yeelight bulb)


Turn on / off (toggle)	Increase brightness	Decrease brightness

Temp (toggle)	Randomly pick	Disco