Mahir Patel
Navoneel Ghosh
Sherry Courington
Yiwen Gu
2021.2.17
We use computer vision techniques to recognize ASL alphabets in a video stream.
Our process starts with background subtraction to perform motion segmentation. We perform median blur with kernel size 3 to remove isolated white pixels from the background. After that we perform gaussian blur and dilation of image further. Performing this step ensures that the palm which would be invisible due to small movements and background subtraction is now filled. After thresholding that image, we merge it with skin detection algorithm so that other stationary objects detected as skin in the background are removed. The final preprocessing step is finding contours in the image and only keeping the one with the largest area. We assume that the user will keep their hands in front of their body so its area will be larger than their face. Finally, we perform multiscale template matching and display that in our UI.
We created a confusion matrix from 97 frames of one of the videos of our gesture detection.
| Truth | |||||||
| "L" | "U" | "C" | "K" | Prediction Sum | Precision | ||
| "L" | 14 | 0 | 0 | 0 | 14 | 100% | |
| Prediction | "U" | 0 | 23 | 0 | 10 | 33 | 69.69% |
| "C" | 3 | 6 | 21 | 6 | 36 | 58.33% | |
| "K" | 0 | 0 | 0 | 14 | 14 | 100% | |
| Truth Sum | 17 | 29 | 21 | 30 | 97 | ||
| Recall | 82.35% | 79.31% | 100% | 46.67% | |||
Accuracy = 74.23%
List your experimental results. Provide examples of input images and output images. If relevant, you may provide images showing any intermediate steps.
If your work involves videos, you can add links here, but also make sure to follow the specific homework instructions on how to submit videos.
The above figure shows the matching results for the 4 templates, i.e. 'L', 'U', 'C', 'K'. Each panel is composed by three images. The left (biggest) one is the frame read from webcam with matched letter showed on its top left corner if found. The top right image is the masked input frames that is used for template matching. The bottom right image in each panel is the template if found.
| L | U | C | K |
|---|---|---|---|
 |
 |
 |
 |
Strengths:
- Our results show that method we adopted is generally successful. We get an accuracy of 0.74
- We scaled our input images progressively during the mathching. As a result, the algorithm will recognize a matched shape as long as the hand is view and not too small.
- We implemented multithreading and the we read frames from the webcam in a separate thread. As a result, our videos in the graphic display looks smoother.
Limitations:
- Our classifier works in normal lightening but if it is too bright or dark, errors will be introduced.
- If the hand in the video is too small (smaller than the template resolution), the mathcing won't be detected.
Potential future work:
- Calculate the orientaion of the templates and objects detected, so that a certain amount of rotation can be allowed.
- Include all the 26 alphabets and/or 10 digits to templates for matching so that we can spell anything.
Fun fact:
- We intentally chose our templates that says "L", "U", "C", "K". We wish all of you and ourselves good luck in the 2021.