/SignPose

A Flask web application game that detects real time Dynamic Sign Language using LSTM Keypoint Detection Model.

Primary LanguagePythonMIT LicenseMIT

SignPose: A Dynamic Sign Language Detection Game

framework libraries models

A LSTM Keypoint Model deployed on Flask that detects real time dynamic sign language (ASL animals) on browser.

Explore AlphaSign, the static version of our sign language game here.

Web Application Home Page

signpose_cow

Animal Dynamic Sign Language Detection

  1. Cow

cow_correct

  1. Elephant

elephant_inference

  1. Butterfly

butterfly_inference

  1. Gorilla

gorilla_inference

  1. Bird

bird_inference

Video Demo of Web App

signpose_video_demo.mp4

Exploratory Data Analysis

Breakdown of Train Test Split using Stratified Sampling (to ensure even distribution of train and test data)

image

LSTM Model Architecture

# LSTM Sequential Model using Keras
model = Sequential()
model.add(LSTM(64, return_sequences=True, activation='relu', input_shape=(30,258))) #each video has input shape of 30 frames of 258 keypoints: X.shape (Handpose + Bodypose)
model.add(Dropout(0.2))
model.add(LSTM(128, return_sequences=True, activation='relu'))
model.add(Dropout(0.2))
model.add(LSTM(64, return_sequences=False, activation='relu')) #next layer is a dense layer so we do not return sequences here
model.add(Dense(64, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(actions.shape[0], activation='softmax'))

# Compile defines the loss function, the optimizer and the metrics. 
model.compile(optimizer='Adam', loss='categorical_crossentropy', metrics=['categorical_accuracy'])

Model Evaluation

Model Training Graphs

  1. Train & Test Categorical Accuracy over epochs

image

  1. Train & Test Loss over epochs (early stopping implemented to prevent overfitting)

image

Confusion Matrix

  1. Train Confusion Matrix

image

  1. Test Confusion Matrix

image

Installation in command prompt

1. Clone Repo

git clone https://github.com/ngzhili/LSTM_Keypoint_Sign_Language_Detector.git

2. Create your virtual environment (venv) using conda

conda create -n <VENV_NAME> python=3.8

3. Activate your virtual environment (venv) using conda

conda activate <VENV_NAME>

4. Install dependencies in venv

pip install -r requirements.txt

5. Change directory to Sign-Language-Image-Recognition

cd <PATH/TO/Sign-Language-Image-Recognition>

6. Run Application in same directory as app.py

flask run