Google Coral on Chronoptics Kea Camera
Requirements
- Camera connected to the internet, so can download updates
- A USB Host DTB (see instructions on how to change to USB host DTB)
To run the examples
- Google Coral device
These are the instructions on how to use the Google Coral on the Chronoptics Kea camera. This is the basic yocto layer to get the google coral stick working on the Chronoptics Kea camera.
Google provides the cross-compiling instructions, https://coral.ai/docs/notes/build-coral/ , unfortunately they use docker to cross-compile. To get a yocto layer working we have done the cross compiling and copied the binary output to the yocto recipe. That way we can get examples demonstrated, however it is not an ideal development environment.
There are two required components
- Edge TPU Runtime (libedgetpu)
- TF Lite Runtime
Two API's
- libcoral API, C++ library
- PyCoral API, Python library
The current method is the Python Wheel's are loaded onto the rootfs, and have to be installed later. This is because of versioning issues with Numpy and Pillow.
- numpy >= 1.19
- pillow >= 8.0.4
- opencv-python >= 4.5.4
Add the clang layer to the sources folder, and checkout the dunfull version
$ cd ~/var-fslc-yocto/sources
$ git clone https://github.com/kraj/meta-clang.git
$ cd meta-clang
$ git checkout dunfell
$ cd ~/var-fslc-yocto/build_xwayland
$ echo 'BBLAYERS += "${BSPDIR}/sources/meta-clang"' >> conf/bblayers.conf
Add the yocto layer to the sources folder
$ cd ~/var-fslc-yocto/sources
$ git clone https://github.com/rzw2/meta-chronoptics-coral
$ cd ~/var-fslc-yocto/build_xwayland
$ echo 'BBLAYERS += "${BSPDIR}/sources/meta-chronoptics-coral"' >> conf/bblayers.conf
Add the extra dependencies
$ echo 'IMAGE_INSTALL_append = "libedgetpu "' >> conf/local.conf
$ echo 'IMAGE_INSTALL_append = "git "' >> conf/local.conf
$ echo 'IMAGE_INSTALL_append = "pycoral "' >> conf/local.conf
$ echo 'IMAGE_INSTALL_append = "tflite-runtime "' >> conf/local.conf
$ echo 'IMAGE_INSTALL_append = "python3-pip "' >> conf/local.conf
$ echo 'IMAGE_INSTALL_append = "python3-opencv "' >> conf/local.conf
$ echo 'IMAGE_INSTALL_append = "portaudio-v19 "' >> conf/local.conf
Now build the image
$ bitbake fsl-image-gui
Flash the SD card
sudo umount /dev/sdX*
$ zcat tmp/deploy/images/kea-c-rev-b/fsl-image-gui-kea-c-rev-b.wic.gz | sudo dd of=/dev/sdX bs=1M conv=fsync
Where X is the SD drive, for example /dev/sda*
Change from the default DTB to the USB Host DTB.
$ cd /media/USERNAME/root/boot
$ sudo rm imx8mq-var-dart-dt8mcustomboard.dtb
$ sudo ln -s kea-c-rev-b-host.dtb imx8mq-var-dart-dt8mcustomboard.dtb
$ sudo rm imx8mq-var-dart-dt8mcustomboard-legacy.dtb
$ sudo ln -s kea-c-rev-b-host.dtb imx8mq-var-dart-dt8mcustomboard-legacy.dtb
Put the SD card into the camera and power the camera to boot.
Log into the camera, either SSH, or use the USB Debug UART, or the HDMI screen with a mouse and keyboard.
In the terminal install the python wheel packages
$ cd /home/root/coral
$ python3 -m pip install tflite_runtime-2.5.0.post1-cp38-cp38-linux_aarch64.whl
$ python3 -m pip install pycoral-2.0.0-cp38-cp38-linux_aarch64.whl
$ ldconfig
Now download an example, we use ( https://github.com/google-coral/examples-camera ).
$ mkdir google-coral && cd google-coral
$ git clone https://github.com/google-coral/examples-camera.git --depth 1
$ cd examples-camera
$ sh download_models.sh
$ cd opencv
To run from the terminal you have to select the display to use
$ DISPLAY=:0 python3 detect.py --camera_idx 1
Cross compiling
The Kea camera uses an IMX8 processor, which is a armv8 instruction set. When you are cross-compiling select aarch64.
Alternativily you can cross compile and copy across to the SD card. These instructions from google go more in depth.
https://coral.ai/docs/notes/build-coral/
TensorFlow Lite Examples
There are examples of using TensorFlow on Kea have been forked from tensorflow are here https://github.com/rzw2/examples The examples that can run on Raspberry Pi should be able to run on Kea as well, however have to change the OpenCV dependency because the version of OpenCV installed with pip depends on qt which fails to launch the OpenCV image viewer.
So for https://github.com/rzw2/examples/tree/master/lite/examples/object_detection/raspberry_pi Follow the instructions to clone the repository.
Now run the setup.sh script and run the example using
DISPLAY=:0 python3 detect.py --enableEdgeTPU --cameraId 1
Examples
Now the Coral stick woks on the camera. You can combine the chronoptics.tof python library (already installed on the camera) to configure the depth camera, get depth frames and pass it to the Google coral for inference. Or use the C++ API's as well.
The Google Coral has examples here: https://coral.ai/examples/ , I recommend using the Yocto SDK to cross compile the examples and load onto the camera.
Other Examples
Other examples of using the google coral
- https://github.com/Namburger/edgetpu-minimal-example This is an example of how to cross compile a C++ program and compile the tf library at the same time. To build
Recommended Development Enviornment
The Google Coral toolchain uses docker which is different from the Kea camera of Yocto. So we recommend developing on a host linux machine, and testing before deploying to the camera. To do use
- Use KeaCamera() on the host Machine, and EmbeddedKeaCamera() on the camera.
- KeaCamera() requires a serial number, while EmbeddedKeaCamera() does not.
- Enable embedded depth processing on KeaCamera() to generate the same depth information.