This repository contains the code accompanying the paper "PIN-a-Boo: Leveraging segmentation and hand skeleton tracking frameworks to discern smartphone PIN entry from a video feed".
-
Determining the region of interest:
- ROI with Padding: Determining the Region of Interest based on the smart phone's location using object detection via a general, pre-trainted YOLOv8 model. Uses padding to avoid changes in windows size at the cost of slightly lower speed.
- ROI without Padding: Same as above but without padding (slightly faster).
- Prior version: Earlier version with some errors.
-
Movement path creation via finger tracking:
- Main Script: Combination of the the prior step (identifying the region of interest with YOLO), finger movement tracking using Mediapipe and drawing of the finger movement path.
- Prior versions: Includes multiple earlier versions (basic mediapipe usage, path creation).
-
Rotation and Transformation:
Includes multiple scripts that try to transform the finger movement path from the camera's perspective into the victim's perspective using a rotation matrix calculated in each frame based on the angles determined via the edges of the phone. Manual adjustments still necessary and not without errors.
- PoC transformation: PoC of a live transformation of the finger path using three rotational angles.
- 2-angle rotation matrix: Calculating a rotational matrix using two angles (smartphone edges).
- Angle calculation: Calculating rotational angles using the long and short edge of the smart phone.
- Other scripts: Contains multiple scripts to compare different line detectors.
-
Decision making:
- Mapping: Maps the normalised finger path (in a list) to a smart phone's number block using clustering.
- Alternative visuals: Produces output in a different visual style.
-
Other scripts:
- Absolute difference: Uses contour detection to show the absolute difference between two frames.
-
Assets:
- Development: Contains pictures and video used for tests during the development.
- Evaluation: Contains videos used for the evaluation.
- YOLOv8: Pre-trained YOLO model by Ultralytics.
The python version and dependencies are quite specific and didn't work on all systems (ARM-Chip Macs).
- Install
pipenv
pip install pipenv
- Navigate to project path
cd /path/to/the/project
- Install dependencies from Pipfile
pipenv install
- Activate the virtual environment
pipenv shell
- Run the file
python folder/whatever.py