View a fuller demo and more background on the project at https://youtu.be/Vi3Li3TkUVY
Semaphore Games uses OpenCV and MediaPipe's Pose detection to perform real-time detection of body landmarks from video input. From there, relative differences are calculated to determine specific positions and translate those into keys and commands sent via keyboard.
For the best experience, you will need a big, well-lit space with a plain background and some combination of an external monitor, external webcam, or extremely good eyesight; standing far enough away from a built-in camera to fit multiple bodies with extended limbs takes significant distance.
The primary way to "type" custom defined keys or key combinations is by extending arms and raising legs at various angles, with some additional recognized motions such as covering the mouth, crossing the arms, jumping, and squatting.
Movements are mapped to keys based on a customizable CSV input file. See data/default.csv
for the default example file. Using the --map
option you can create and load in your own key mappings for different games.
Example common usage:
python play.py -f 1 -l 1 -s 2 -r 1 -m pong.csv
- Flip the camera, draw body
landmarks, split the screen for two players, record the results to a timestamped AVI, and use the pong.csv
map for input controls.
By default, extended arm angles snap to the closest 45 degrees, and leg angles are set to 18 degrees off of vertical (i.e. 18 for 'low kick' and 36 for 'high kick'). Modify LEG_EXTEND_ANGLE
and the resulting angles in mapped leg extensions to update this sensitivity.
An input file for key mapping should follow the format in default.csv
, with the following columns:
- 0: player number, starting at 1
- 1: body part or action, such as
left arm
orjump
; full list below - 2: value for part or action: degrees such as 45 for arms, or 0/1/2 for legs
- 3: key or keys to press, separated by spaces
- 4: user-defined action description, such as
accelerate
orlow punch
The first row of each mapping file is skipped; it can be used for notes such as identifying or linking to the relevant game.
The full set of parts/actions available are as follows:
left arm
: takes values rounded to 45 degrees for fully extended arm; see below for degreesright arm
: same as aboveleft leg
: takes values of 0 (low straight leg lift), 1 (high leg lift), or 2 (leg lift, bent at knee)right leg
: same as abovecrossed arms
: only takes the value1
; both palms touching opposite elbowsjump
: only takes the value1
; hips quickly rising then fallingsquat
: only takes the value1
; both thighs parallel to ground, overrides leg liftsmouth
: only takes the value1
; cover mouth with both palms
Right arm degrees: 90 straight up to 0 right to -90 down by side Left arm degrees: 90 straight up to 180 left to 269... -> -90 down by side
Running on latest MacOS from Terminal, toggle the following for keyboard access:
System Settings -> Privacy & Security -> Accessibility -> Terminal -> slide to allow
For Mac, this uses a custom keyboard library imported locally to correct key mappings.