/video-analytics-serving

Primary LanguagePythonBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

Video Analytics Serving

| Getting Started | Documentation | Reference Guides | Related Links |

Video Analytics Serving is a python package and microservice for deploying optimized media analytics pipelines. It supports pipelines defined in GStreamer* or FFmpeg* and provides APIs to discover, start, stop, customize and monitor pipeline execution. Video Analytics Serving is based on OpenVINO™ Toolkit DL Streamer and FFmpeg Video Analytics.

Features Include:
Customizable Media Analytics Containers Scripts and dockerfiles to build and run container images with the required dependencies for hardware optimized media analytics pipelines.
No-Code Pipeline Definitions and Templates JSON based definition files, a flexible way for developers to define and parameterize pipelines while abstracting the low level details from their users.
Deep Learning Model Integration A simple way to package and reference OpenVINO™ based models in pipeline definitions. The precision of a model can be auto-selected at runtime based on the chosen inference device.
Video Analytics Serving Python API A python module to discover, start, stop, customize and monitor pipelines based on their no-code definitions.
Video Analytics Serving Microservice                                                                       A RESTful microservice providing endpoints and APIs matching the functionality of the python module.

IMPORTANT: Video Analytics Serving is provided as a sample. It is not intended to be deployed into production environments without modification. Developers deploying Video Analytics Serving should review it against their production requirements.

Getting Started

The sample microservice includes three media analytics pipelines.
object_detection Detect and label objects such as bottles and bicycles.
emotion_recognition Detect the emotions of a person within a video stream.
audio_detection Analyze audio streams for events such as breaking glass or barking dogs.

Prerequisites
Docker Video Analytics Serving requires Docker for it's build, development, and runtime environments. Please install the latest for your platform. Docker.
bash Video Analytics Serving's build and run scripts require bash and have been tested on systems using versions greater than or equal to: GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu). Most users shouldn't need to update their version but if you run into issues please install the latest for your platform. Instructions for macOS®* users here.
curl The samples below use the curl command line program to issue standard HTTP requests to the microservice. Please install the latest for your platform. Note: any other tool or utility that can issue standard HTTP requests can be used in place of curl.

Building the Microservice

Build the sample microservice with the following command:

./docker/build.sh

The script will automatically include the sample models, pipelines and required dependencies.

Note: When running this command for the first time, the default base image for Video Analytics Serving will take a long time to build (likely over an hour). For instructions on how to re-use pre-built base images to speed up the build time please see the following documentation.

To verify the build succeeded execute the following command:

docker images video-analytics-serving-gstreamer:latest

Expected output:

REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
video-analytics-serving-gstreamer   latest              f51f2695639f        2 minutes ago          1.39GB

Running the Microservice

Start the sample microservice with the following command:

./docker/run.sh -v /tmp:/tmp

This script issues a standard docker run command to launch the container, run a Tornado based web service on port 8080, and mount the /tmp folder. The /tmp folder is mounted to share sample results with the host and is optional in actual deployments.

Expected output:

{"levelname": "INFO", "asctime": "2020-08-06 12:37:12,139", "message": "=================", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:12,139", "message": "Loading Pipelines", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:12,139", "message": "=================", "module": "pipeline_manager"}
(gst-plugin-scanner:14): GStreamer-WARNING **: 12:37:12.476: Failed to load plugin '/root/gst-video-analytics/build/intel64/Release/lib/libvasot.so': libopencv_video.so.4.4: cannot open shared object file: No such file or directory
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,207", "message": "FFmpeg Pipelines Not Enabled: ffmpeg not installed\n", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,208", "message": "Loading Pipelines from Config Path /home/video-analytics-serving/pipelines", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,223", "message": "Loading Pipeline: audio_detection version: 1 type: GStreamer from /home/video-analytics-serving/pipelines/audio_detection/1/pipeline.json", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,230", "message": "Loading Pipeline: object_detection version: 1 type: GStreamer from /home/video-analytics-serving/pipelines/object_detection/1/pipeline.json", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,240", "message": "Loading Pipeline: emotion_recognition version: 1 type: GStreamer from /home/video-analytics-serving/pipelines/emotion_recognition/1/pipeline.json", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,241", "message": "===========================", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,241", "message": "Completed Loading Pipelines", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,241", "message": "===========================", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,333", "message": "Starting Tornado Server on port: 8080", "module": "__main__"}

Detecting Objects in a Video

Example Request:

Endpoint Verb Request Response
pipelines/object_detection/1 POST 
JSON
{
  "source": {
    "uri": "https://example.mp4",
    "type": "uri"
  },
  "destination": {
    "type": "file",
    "path": "/tmp/results_objects.txt",
    "format": "json-lines"
  }
}
 200

Pipeline Instance Id

Curl Command:

Start a new shell and execute the following command to issue an HTTP POST request, start a pipeline and analyze a sample video.

curl localhost:8080/pipelines/object_detection/1 -X POST -H \
'Content-Type: application/json' -d \
'{ 
  "source": {
    "uri": "https://github.com/intel-iot-devkit/sample-videos/blob/master/bottle-detection.mp4?raw=true",
    "type": "uri"
  },
  "destination": {
    "type": "file",
    "path": "/tmp/results_objects.txt",
    "format": "json-lines"
  }
}'

Detection Results:

To view incremental results, execute the following command from the shell.

tail -f /tmp/results_objects.txt

As the video is being analyzed and as objects appear and disappear you will see detection results in the output.

Expected Output:

{"objects":[{"detection":{"bounding_box":{"x_max":0.9022353887557983,"x_min":0.7940621376037598,"y_max":0.8917602300643921,"y_min":0.30396613478660583},"confidence":0.7093080282211304,"label":"bottle","label_id":5},"h":212,"roi_type":"bottle","w":69,"x":508,"y":109}],"resolution":{"height":360,"width":640},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/bottle-detection.mp4?raw=true","timestamp":39553072625}

After pretty-printing:

{
  "objects": [
    {
      "detection": {
        "bounding_box": {
          "x_max": 0.9022353887557983,
          "x_min": 0.7940621376037598,
          "y_max": 0.8917602300643921,
          "y_min": 0.30396613478660583
        },
        "confidence": 0.7093080282211304,
        "label": "bottle",
        "label_id": 5
      },
      "h": 212,
      "roi_type": "bottle",
      "w": 69,
      "x": 508,
      "y": 109
    }
  ],
  "resolution": {
    "height": 360,
    "width": 640
  },
  "source": "https://github.com/intel-iot-devkit/sample-videos/blob/master/bottle-detection.mp4?raw=true",
  "timestamp": 39553072625
}

More Examples

Emotion Recognition

Recognizing Emotions in a Video

Example Request:

Endpoint Verb Request Response
/pipelines/emotion_recognition/1 POST 
JSON
{
  "source": {
    "uri": "https://example.mp4",
    "type": "uri"
  },
  "destination": {
    "type": "file",
    "path": "/tmp/results_emotions.txt",
    "format": "json-lines"
  }
}
 200

Pipeline Instance Id

Curl Command:

Start a new shell and execute the following command to issue an HTTP POST request, start a pipeline and analyze a sample video.

curl localhost:8080/pipelines/object_detection/1 -X POST -H \
'Content-Type: application/json' -d \
'{ 
  "source": {
    "uri": "https://github.com/intel-iot-devkit/sample-videos/blob/master/head-pose-face-detection-male.mp4?raw=true",
    "type": "uri"
  },
  "destination": {
    "type": "file",
    "path": "/tmp/results_emotions.txt",
    "format": "json-lines"
  }
}'

Detection Results:

To view incremental results, execute the following command from the shell.

tail -f /tmp/results_emotions.txt

As the video is being analyzed and as the person's emotions appear and disappear you will see recognition results in the output.

Expected Output:

{"objects":[{"detection":{"bounding_box":{"x_max":0.567557156085968,"x_min":0.42375022172927856,"y_max":0.5346322059631348,"y_min":0.15673652291297913},"confidence":0.9999996423721313,"label":"face","label_id":1},"emotion":{"label":"neutral","model":{"name":"0003_EmoNet_ResNet10"}},"h":163,"roi_type":"face","w":111,"x":325,"y":68}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/head-pose-face-detection-male.mp4?raw=true","timestamp":13333333333}

After pretty-printing:

{
  "objects": [
    {
      "detection": {
        "bounding_box": {
          "x_max": 0.567557156085968,
          "x_min": 0.42375022172927856,
          "y_max": 0.5346322059631348,
          "y_min": 0.15673652291297913
        },
        "confidence": 0.9999996423721313,
        "label": "face",
        "label_id": 1
      },
      "emotion": {
        "label": "neutral",
        "model": {
          "name": "0003_EmoNet_ResNet10"
        }
      },
      "h": 163,
      "roi_type": "face",
      "w": 111,
      "x": 325,
      "y": 68
    }
  ],
  "resolution": {
    "height": 432,
    "width": 768
  },
  "source": "https://github.com/intel-iot-devkit/sample-videos/blob/master/head-pose-face-detection-male.mp4?raw=true",
  "timestamp": 13333333333
}
Audio Event Detection

Detecting Audio Events in an Audio Recording

Example Request:

Endpoint Verb Request Response
/pipelines/audio_detection/1 POST 
JSON
{
  "source": {
    "uri": "https://example.wav",
    "type": "uri"
  },
  "destination": {
    "type": "file",
    "path": "/tmp/results_audio_events.txt",
    "format": "json-lines"
  }
}
 200

Pipeline Instance Id

Curl Command:

Start a new shell and execute the following command to issue an HTTP POST request, start a pipeline and analyze a sample audio.

curl localhost:8080/pipelines/object_detection/1 -X POST -H \
'Content-Type: application/json' -d \
'{ 
  "source": {
    "uri": "https://github.com/opencv/gst-video-analytics/blob/preview/audio-detect/samples/gst_launch/audio_detect/how_are_you_doing.wav?raw=true",
    "type": "uri"
  },
  "destination": {
    "type": "file",
    "path": "/tmp/results_audio_events.txt",
    "format": "json-lines"
  }
}'

Detection Results:

To view incremental results, execute the following command from the shell.

tail -f /tmp/results_audio_events.txt

As the audio is being analyzed and as events start and stop you will see detection results in the output.

Expected Output:

{"channels":1,"events":[{"detection":{"confidence":1.0,"label":"Speech","label_id":53,"segment":{"end_timestamp":2200000000,"start_timestamp":1200000000}},"end_timestamp":2200000000,"event_type":"Speech","start_timestamp":1200000000}],"rate":16000}

After pretty-printing:

{
  "channels": 1,
  "events": [
    {
      "detection": {
        "confidence": 1,
        "label": "Speech",
        "label_id": 53,
        "segment": {
          "end_timestamp": 2200000000,
          "start_timestamp": 1200000000
        }
      },
      "end_timestamp": 2200000000,
      "event_type": "Speech",
      "start_timestamp": 1200000000
    }
  ],
  "rate": 16000
}

Further Reading

Documentation Reference Guides
- Defining Media Analytics Pipelines
- Building Video Analytics Serving
- Running Video Analytics Serving		 - Video Analytics Serving Architecture Diagram
- Microservice Endpoints
- Build Script Reference
- Run Script Reference

Related Links

Media Frameworks Media Analytics Samples and Reference Designs
- GStreamer*
- GStreamer* Overview
- FFmpeg*		 - OpenVINO™ Toolkit
- OpenVINO™ Toolkit DL Streamer
- FFmpeg* Video Analytics		 - Open Visual Cloud Smart City Sample
- Open Visual Cloud Ad-Insertion Sample
- Edge Insights for Retail

* Other names and brands may be claimed as the property of others.