/air_control

Primary LanguagePython

Air control

A virtual mouse based on hand movement using webcam, google MediaPipe, python mouse and openCV in python

Before testing the virtual mouse, make sure you have saved and closed your important programs, the ghost click👻 can be very dangerous, use with full caution❗:shipit:

Feature

  1. Place right hand index finger in pointer box and it will translate your mouse position to your finger location
    Pointer GIF
  2. Place left hand index and thumb fingers left click box, and then make it close the distance and release it to do a left click, do it twice to double left click
    Left click GIF
  3. Place left hand index and thumb fingers right click box, and then make it close the distance and release it to do a right click
    Right click GIF
  4. Place left hand index and thumb finger in left or right click box, and then make it close the distance and hold it to do a mouse drag, and release the hold it to drop
    Drag GIF

Notes

  • Only able to detect a pair of hand per screen (if you show 2 right hand on screen, it will be recognized as one hand, so the coordinate will jump from one hand to another)
  • Your clothes and background might be classified as a "Hand" too (may lead to ghost click👻), so better set the IS_CLICK_HAND_ON and IS_POINTER_HAND_ON to False to test your environment first
  • Only support 1 screen at one time, at 16:9 resolution

Prerequisite

  1. Python >= 3.7.9
  2. Webcam

How to install

  1. Install python (you can refer to Python site)
  2. (Optional) Initiate venv by running python -m venv ac_env
  3. Install library via pip from requirements.txt by running python -m pip install requirements.txt
  4. Adjust the settings located at the ./settings/ folder (the IS_CLICK_HAND_ON and IS_POINTER_HAND_ON value, you need to set the value to True to move your mouse, or keep it to False if you want to check for ghost click first)
  5. Run with python main.py

Notes for installation

Windows

You are good to go by just following the installation, no need for additional works

Linux

Makesure to use XOrg for the display server (Ubuntu 21.04 use wayland, you need to change it first) to enable the mouse library

Other OS (Including Mac)

Not yet tested

Stabilizer

Because the MediaPipe usually return unstable coordinate of a finger tip, and mouse is usually used with high stability, a stabilizer function is needed. This project have 4 options of stabilizers, you can choose which one you want to use and other value in ./settings/stabilizer_settings.py. The stabilizer function will have 3 input parameter, the finger_tip_x and finger_tip_y list with MINIMUM_TIP_POINT_TO_CALCULATE_CENTROID numbers of points, and last_position (the last tip coordinate). finger_tip_x and finger_tip_y list use FIFO mechanism to save value, updated each frame, with maximum number of MAX_TIP_POINTS values.

Calculate Centroid 1

This one is created to keep the persistance of the tip when the finger stopped, but keep it moving when finger moves

  1. For the first MINIMUM_TIP_POINT_TO_CALCULATE_CENTROID coordinate, directly move to the current finger coordinate
  2. If the finger_tip_x list already have MINIMUM_TIP_POINT_TO_CALCULATE_CENTROID number of coordinates, then calculate the centroid
  3. Calculate the distance between next coordinate centroid and the last position coordinate (last_position) if the distance is above MAXIMUM_CENTROID_1_DISTANCE, then next coordinate is the centroid, else the next coordinate is same with last coordinate

Calculate Centroid 2

This one is created to keep the persistance of the tip when the finger stopped, but keep it moving when the finger moves, faster than Calculate Centroid 1, but less persistance

  1. For the first MINIMUM_TIP_POINT_TO_CALCULATE_CENTROID coordinate, directly move to the current finger coordinate
  2. Calculate centroid from saved point in finger_tip_x and finger_tip_y list, except the last value (current finger position)
  3. Calculate the distance between centroid and current finger coordinate
  4. If the distance passthrough MAXIMUM_CENTROID_2_DISTANCE, then use the current finger coordinate as next coordinate, also refill all value in finger_tip_x and finger_tip_y with current finger coordinate
  5. If the distance doesn't passthrough, then use the last_position value

Smooth Move

Source : Youtube video
Slowly move the point by dividing distance between last coordinate and current finger coordinate with SMOOTHING_POINT, and add it to the last coordinate so the next coordinate will not directly jump to the current finger coordinate, but slowly moving the point towards the current finger coordinate

Smooth Move with Minimum Angle

Same with Smooth Move, but it will use the last two coordinate and current finger coordinate to calculate the angle between the three point, if it passthrough the MINIMUM_TIP_POINT_TO_CALCULATE_ANGLE then the Slow Move stabilizer calculated, else it will directly jump to the current finger coordinate

No Stabilizer

Directly jump to current finger location