Tensorflow Lite Image Analysis using Camera4Kivy
Uses Tensoflow Lite to classify objects in the image stream, classified objects are boxed and labeled in the Preview.
Available on some of the usual platforms.
This example is based on a Tensorflow Lite Object Detection Example. This example was trained on the COCO dataset.
This example was chosen because it was trained on a large dataset and thus slow to infer objects. On newish Intel based machines annotation frame rates can be of the order of a few fps with additional latency for first recognition, video frame rates are normal. Windows machines were about 3x slower than Linux based machines, on a 4 year old Windows i3 machine the annotation rate is 1/10 fps.
If you want better performace consider a Coral Accelerator. This gave me very approximately an order of magnitude speed improvement.
As of 2021/12/13 the Python tflite_runtime.whl is not available for any of these: Python 3.10, MacOS on M1, MacOS Monterey, iOS, x86_32, or Android. If you use any of these, stop now. If you have a problem with this talk to Google.
A recipe for tflite-runtime on Android is on my TODO list.
Tensorflow Image analysis has four distinct components:
-
The tensorflow lite model, formated to include tensor labels. Created for example with Model Maker and here.
-
The Python
tflite-runtimepackage. This performs inference based on the model and the input image. This package is available from Google on most desktops. -
The model interface, specific to the tflite model in this case
object_detection/object_detector.py. This encodes the model input, executes the runtime, and decodes the output. -
The Camera4Kivy interface classifyobject.py. This passes the image to the model interface, and annotates the output to the Preview.
This example depends on Camera4Kivy. Depending on the platform you may need to install a camera provider.
pip3 install numpy opencv-python camera4kivy
The currently available version of tflite-runtime maybe different to the one used on the next line.
pip3 install --index-url https://google-coral.github.io/py-repo/tflite-runtime==2.5.0.post1
If you use a Coral Accelerator set enable_edgetpu = True in classifyobject.py.