TensorFlow BodyPix (TF BodyPix)

A Python implementation of body-pix.

Goals of this project is:

Python library, making it easy to integrate the BodyPix model
CLI with limited functionality, mostly for demonstration purpose

Prerequisits

Python 3.7+

Install

Install with all dependencies:

pip install tf-bodypix[all]

Install with minimal or no dependencies:

pip install tf-bodypix

Extras are provided to make it easier to provide or exclude dependencies when using this project as a library:

extra name	description
tf	TensorFlow (required). But you may use your own build.
tfjs	TensorFlow JS Model support, using tfjs-graph-converter
tflite	tflite-runtime
image	Image loading via Pillow, required by the CLI.
video	Video support via OpenCV
webcam	Webcam support via OpenCV and pyfakewebcam
all	All of the libraries (except `tflite-runtime`)

Python API

from pathlib import Path
import tensorflow as tf
from tf_bodypix.api import download_model, load_model, BodyPixModelPaths

# setup input and output paths
output_path = Path('./data/example-output')
output_path.mkdir(parents=True, exist_ok=True)
input_url = (
    'https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1'
)
local_input_path = tf.keras.utils.get_file(origin=input_url)

# load model (once)
bodypix_model = load_model(download_model(
    BodyPixModelPaths.MOBILENET_FLOAT_50_STRIDE_16
))

# get prediction result
image = tf.keras.preprocessing.image.load_img(local_input_path)
image_array = tf.keras.preprocessing.image.img_to_array(image)
result = bodypix_model.predict_single(image_array)

# simple mask
mask = result.get_mask(threshold=0.75)
tf.keras.preprocessing.image.save_img(
    f'{output_path}/output-mask.jpg',
    mask
)

# colored mask (separate colour for each body part)
colored_mask = result.get_colored_part_mask(mask)
tf.keras.preprocessing.image.save_img(
    f'{output_path}/output-colored-mask.jpg',
    colored_mask
)

# poses
from tf_bodypix.draw import draw_poses  # utility function using OpenCV

poses = result.get_poses()
image_with_poses = draw_poses(
    image_array.copy(),  # create a copy to ensure we are not modifing the source image
    poses,
    keypoints_color=(255, 100, 100),
    skeleton_color=(100, 100, 255)
)
tf.keras.preprocessing.image.save_img(
    f'{output_path}/output-poses.jpg',
    image_with_poses
)

CLI

CLI Help

python -m tf_bodypix --help

python -m tf_bodypix <sub command> --help

List Available Models

python -m tf_bodypix list-models

The result will be a list of all of the bodypix TensorFlow JS models available in the tfjs-models bucket.

Those URLs can be passed as the --model-path arguments below, or to the download_model method of the Python API.

The CLI will download and cache the model from the provided path. If no --model-path is provided, it will use a default model (mobilenet).

To list TensorFlow Lite models instead:

python -m tf_bodypix list-tflite-models

Inputs and Outputs

Most commands will work with inputs (source) and outputs.

The source path can be specified via the --source parameter.

The following inputs are supported:

type	description
image	Static image (e.g. `.png`)
video	Video (e.g. `.mp4`)
webcam	Linux Webcam (`/dev/videoN` or `webcam:0`)

If the source path points to an external file (e.g. https://), then it will be downloaded and locally cached.

The output path can be specified via --output, unless --show-output is used.

The following outpus are supported:

type	description
image_writer	Write to a static image (e.g. `.png`)
v4l2	Linux Virtual Webcam (`/dev/videoN`)
window	Display a window (by using `--show-output`)

Example commands

Creating a simple body mask

python -m tf_bodypix \
    draw-mask \
    --source \
    "https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1" \
    --show-output \
    --threshold=0.75

Image Source: Serious black businesswoman sitting at desk in office

Add the mask over the original image using `--mask-alpha`

python -m tf_bodypix \
    draw-mask \
    --source \
    "https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1" \
    --show-output \
    --threshold=0.75 \
    --mask-alpha=0.5

Image Source: Serious black businesswoman sitting at desk in office

Colorize the body mask depending on the body part

python -m tf_bodypix \
    draw-mask \
    --source \
    "https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1" \
    --show-output \
    --threshold=0.75 \
    --mask-alpha=0.5 \
    --colored

Image Source: Serious black businesswoman sitting at desk in office

Additionally select the body parts

python -m tf_bodypix \
    draw-mask \
    --source \
    "https://www.dropbox.com/s/7tsaqgdp149d8aj/serious-black-businesswoman-sitting-at-desk-in-office-5669603.jpg?dl=1" \
    --show-output \
    --threshold=0.75 \
    --mask-alpha=0.5 \
    --parts left_face right_face \
    --colored

Image Source: Serious black businesswoman sitting at desk in office

Add mask overlay to a video

python -m tf_bodypix \
    draw-mask \
    --source \
    "https://www.dropbox.com/s/s7jga3f0dreavlb/video-of-a-man-laughing-and-happy-1608393-360p.mp4?dl=1" \
    --show-output \
    --threshold=0.75 \
    --mask-alpha=0.5 \
    --colored

Video Source: Video Of A Man Laughing And Happy

Add pose overlay to a video

python -m tf_bodypix \
    draw-pose \
    --source \
    "https://www.dropbox.com/s/pv5v8dkpj5wung7/an-old-man-doing-a-tai-chi-exercise-2882799-360p.mp4?dl=1" \
    --show-output \
    --threshold=0.75

Blur background of a video

python -m tf_bodypix \
    blur-background \
    --source \
    "https://www.dropbox.com/s/s7jga3f0dreavlb/video-of-a-man-laughing-and-happy-1608393-360p.mp4?dl=1" \
    --show-output \
    --threshold=0.75 \
    --mask-blur=5 \
    --background-blur=20

Video Source: Video Of A Man Laughing And Happy

Replace the background of a video

python -m tf_bodypix \
    replace-background \
    --source \
    "https://www.dropbox.com/s/s7jga3f0dreavlb/video-of-a-man-laughing-and-happy-1608393-360p.mp4?dl=1" \
    --background \
    "https://www.dropbox.com/s/b22ss59j6pp83zy/brown-landscape-under-grey-sky-3244513.jpg?dl=1" \
    --show-output \
    --threshold=0.75 \
    --mask-blur=5

Video Source: Video Of A Man Laughing And Happy

Background: Brown Landscape Under Grey Sky

Capture Webcam and adding mask overlay

python -m tf_bodypix \
    draw-mask \
    --source webcam:0 \
    --show-output \
    --threshold=0.75 \
    --mask-alpha=0.5 \
    --colored

Capture Webcam and adding mask overlay, writing to v4l2loopback device

(replace /dev/videoN with the actual virtual video device)

python -m tf_bodypix \
    draw-mask \
    --source webcam:0 \
    --output /dev/videoN \
    --threshold=0.75 \
    --mask-alpha=0.5 \
    --colored

Capture Webcam and blur background, writing to v4l2loopback device

(replace /dev/videoN with the actual virtual video device)

python -m tf_bodypix \
    blur-background \
    --source webcam:0 \
    --background-blur 20 \
    --output /dev/videoN \
    --threshold=0.75

Capture Webcam and replace background, writing to v4l2loopback device

(replace /dev/videoN with the actual virtual video device)

python -m tf_bodypix \
    replace-background \
    --source webcam:0 \
    --background \
    "https://www.dropbox.com/s/b22ss59j6pp83zy/brown-landscape-under-grey-sky-3244513.jpg?dl=1" \
    --threshold=0.75 \
    --output /dev/videoN

Background: Brown Landscape Under Grey Sky

TensorFlow Lite Model support (experimental)

The model path may also point to a TensorFlow Lite model (.tflite extension). Whether that actually improves performance may depend on the platform and available hardware.

You could convert one of the available TensorFlow JS models to TensorFlow Lite using the following command:

python -m tf_bodypix \
    convert-to-tflite \
    --model-path \
    "https://storage.googleapis.com/tfjs-models/savedmodel/bodypix/mobilenet/float/075/model-stride16.json" \
    --optimize \
    --quantization-type=float16 \
    --output-model-file "./mobilenet-float-multiplier-075-stride16-float16.tflite"

The above command is provided for convenience. You may use alternative methods depending on your preference and requirements.

Relevant links:

TensorFlow Lite Runtime support (experimental)

This project can also be used with tflite-runtime instead of full TensorFlow (e.g. by using the tflite extra). However, TensorFlow Lite converter would require full TensorFlow. In order to avoid it, one needs to use a TensorFlow Lite model (see previous section).

Docker Usage

You could also use the Docker image if you prefer. The entrypoint will by default delegate to the CLI, except for python or bash commands.

# pull latest image (you may also use tags)
docker pull de4code/tf-bodypix

# mount real and virtual webcam devices on linux
docker run --rm \
    --device /dev/video0 \
    --device /dev/video2 \
    de4code/tf-bodypix \
    blur-background \
    --source /dev/video0 \
    --output /dev/video2 \
    --background-blur 20 \
    --threshold=0.75

# mount x11 display on linux
docker run --rm \
    --net=host \
    --volume /tmp/.X11-unix:/tmp/.X11-unix \
    --volume ${HOME}/.Xauthority:/root/.Xauthority \
    --env DISPLAY \
    de4code/tf-bodypix \
    replace-background \
    --source \
    "https://www.dropbox.com/s/s7jga3f0dreavlb/video-of-a-man-laughing-and-happy-1608393-360p.mp4?dl=1" \
    --background \
    "https://www.dropbox.com/s/b22ss59j6pp83zy/brown-landscape-under-grey-sky-3244513.jpg?dl=1" \
    --show-output \
    --threshold=0.75 \
    --mask-blur=5

Example Media

Here are a few example media files you could try.

Images:

Videos:

Background:

Brown Landscape Under Grey Sky (Source)

Experimental Downstream Projects

Layered Vision is an experimental project using the tf-bodypix Python API.

Acknowledgements

Original TensorFlow JS Implementation of BodyPix
Linux-Fake-Background-Webcam, an implementation of the blog post describing using the TensorFlow JS implementation with Python via a Socket API.
tfjs-to-tf for providing an easy way to convert TensorFlow JS models
virtual_webcam_background for a great pure Python implementation

de-code/python-tf-bodypix

TensorFlow BodyPix (TF BodyPix)

Prerequisits

Install

Python API

CLI

CLI Help

List Available Models

Inputs and Outputs

Example commands

Creating a simple body mask

Add the mask over the original image using --mask-alpha

Colorize the body mask depending on the body part

Additionally select the body parts

Add mask overlay to a video

Add pose overlay to a video

Blur background of a video

Replace the background of a video

Capture Webcam and adding mask overlay

Capture Webcam and adding mask overlay, writing to v4l2loopback device

Capture Webcam and blur background, writing to v4l2loopback device

Capture Webcam and replace background, writing to v4l2loopback device

TensorFlow Lite Model support (experimental)

TensorFlow Lite Runtime support (experimental)

Docker Usage

Example Media

Experimental Downstream Projects

Acknowledgements

Add the mask over the original image using `--mask-alpha`