GStreamer Neural Network Inference demo for NXP i.MX Processors
===============================================================
ABOUT THIS SOFTWARE
-------------------
This software demonstarates GStreamer Neural Network Inference for NXP i.MX processors.
It uses an object detection model and a pose estimation model published in the internet.
Binary distribution includes demo scripts, gstreamer plugin, TensorFlow Lite models and documents.
Demo scripts:
/usr/bin/gstnninferencedemo-mobilenet-ssd-camera
/usr/bin/gstnninferencedemo-mobilenet-ssd-video
/usr/bin/gstnninferencedemo-posenet-camera
/usr/bin/gstnninferencedemo-posenet-video
GStreamer plugin:
/usr/lib/gstreamer-1.0/libgstnninferencedemo.so
TensorFlow Lite models and documents
/usr/share/gstnninferencedemo/*
PLATFORM
--------
This software is tested with below environment:
1) BSP
- i.MX Linux BSP 5.4.47-2.2.0
- i.MX Linux BSP 5.10.x
2) Boards
- i.MX 8M Plus EVK (CPU rev 1.1)
- i.MX 8M Mini EVK
3) Camera
- MIPI camera
BUILD
-----
The commands to build this software in x86_64 linux host PC are below:
$ git clone {this-git}
$ cd eiq-example-apps
$ git submodule init
$ git submodule update
$ cd google-coral/
$ ./download_dependencies.sh
$ cd edgetpu
/* Setup the cross-build env with i.MX sdk. For example, source /opt/fsl-imx-internal-xwayland/5.10-gatesgarth/environment-setup-cortexa53-crypto-poky-linux. */
$ source <script with environment variables>
$ ./autogen.sh
$ ./configure --host=aarch64-poky-linux
$ make
/* If build successfully, libgooglecoraledgetpuposenet.a will be generated under google-coral/edgetpu/.libs/. */
$ cd eiq-example-apps
/* Setup the cross-build env with i.MX sdk. For example, source /opt/fsl-imx-internal-xwayland/5.10-gatesgarth/environment-setup-cortexa53-crypto-poky-linux. */
$ source <script with environment variables>
$ ./autogen.sh
$ ./configure --host=aarch64-poky-linux CPPFLAGS="-I$(pwd)/google-coral/edgetpu/src/cpp/" LDFLAGS="$(pwd)/google-coral/edgetpu/.libs/libgooglecoraledgetpuposenet.a"
$ make
/* If build successfully, libgstnninferencedemo.so will be generated under eiq-example-apps/src/.libs/. */
INSTALL
-----
1) Install this package to imx-image-full rootfs, because it depends on OpenCV shared libs.
2) Copy demo scripts gstnninferencedemo-mobilenet-ssd-camera, gstnninferencedemo-mobilenet-ssd-video, gstnninferencedemo-posenet-camera and gstnninferencedemo-posenet-video to /usr/bin in root filesystem.
# cp eiq-example-apps/src/gstnninferencedemo-mobilenet-ssd-camera {rootfs}/usr/bin/
# cp eiq-example-apps/src/gstnninferencedemo-mobilenet-ssd-video {rootfs}/usr/bin/
# cp eiq-example-apps/src/gstnninferencedemo-posenet-camera {rootfs}/usr/bin/
# cp eiq-example-apps/src/gstnninferencedemo-posenet-video {rootfs}/usr/bin/
3) Copy the generated gstreamer plugin library libgstnninferencedemo.so to /usr/lib/gstreamer-1.0 in root filesystem.
# cp libgstnninferencedemo.so {rootfs}/usr/lib/gstreamer-1.0/
4) Copy TFLite models to /usr/share/gstnninferencedemo/google-coral in root filesystem.
# mkdir -p {rootfs}/usr/share/gstnninferencedemo/google-coral/examples-camera
# mkdir -p {rootfs}/usr/share/gstnninferencedemo/google-coral/project-posenet
# cp google-coral/examples-camera/all_models/coco_labels.txt {rootfs}/usr/share/gstnninferencedemo/google-coral/examples-camera/
# cp google-coral/examples-camera/all_models/mobilenet_ssd_v2_coco_quant_postprocess.tflite {rootfs}/usr/share/gstnninferencedemo/google-coral/examples-camera/
# cp google-coral/project-posenet/models/posenet_mobilenet_v1_075_353_481_quant_decoder.tflite {rootfs}/usr/share/gstnninferencedemo/google-coral/project-posenet/
USAGE
-----
1) Boot the i.MX EVK board and login to Linux as root
2) Run below demo scripts
- Object detection with video stream (720p30 video is recommended.)
# gstnninferencedemo-mobilenet-ssd-video </path/to/video_file>
- Object detection with camera stream
(For example, MIPI camera is plugged in CSI1 MIPI port on i.MX 8M Plus EVK. /dev/video1 is used.)
# gstnninferencedemo-mobilenet-ssd-video [/dev/video?]
- Pose estimation with video stream (720p30 video is recommended.)
# gstnninferencedemo-posenet-video </path/to/video_file>
- Pose estimation with camera stream
(For example, MIPI camera is plugged in CSI1 MIPI port on i.MX 8M Plus EVK. /dev/video1 is used.)
# gstnninferencedemo-posenet-video [/dev/video?]
3) If you want to play more, modify demo scripts freely!
ACKNOWLEDGEMENTS
----------------
I'm deeply grateful to teams/communities developing great software below:
This software was written with reference to:
- imx-gst1.0-plugin <https://source.codeaurora.org/external/imx/imx-gst1.0-plugin>
This software is inspired by the work from below:
- google-coral/examples-camera <https://github.com/google-coral/examples-camera>
- google-coral/project-posenet <https://github.com/google-coral/project-posenet>
- google-coral/edgetpu <https://github.com/google-coral/edgetpu>
LICENSE
----------------
This software is licensed under LGPL-2.0+
DISCLAIMER
----------
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO
THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT
SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR
ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE
USE OR PERFORMANCE OF THIS SOFTWARE.