The purpose of this project is to make a custom micropython firmware that installs tensorflow lite for micro controllers and allows for experimentation.
This project is a micropython module built using the USER_C_MODULES extension mechanism. There are several modules:
- microlite
- ulab
- modcamera (for the person_detection example)
There are 4 top level git submodules:
- tensorflow lite micro
- micropython
- ulab
- tflm_esp_kernels
tflite-micro sources are generated within the microlite module at build time using the tensorflow lite example generator.
The microlite module has several types:
- tensor
- interpreter
- audio_frontend (used for the micro_speech example)
| Build Type | Status |
|---|---|
| ESP32 |  |
| ESP32 S3 |  |
| RP2 |  |
| STM32 | Doesn't Work |
| UNIX |  |
The latest firmware can be downloaded from the applicable build above (in the Status section).
- Click on build status link.
- Click on the latest green build
- Review the available artifacts for download
You do need to be careful to get the proper firmware for your board. If your board is not currently being built please file an issue and it can be added.
Also be sure that you are getting the most recent boards build from the main branch. There are builds from other feature branches but many, especially those related to optional ops, are broken even if it looks like the build worked properly.
The STM32 port works for hello_world now.
At the moment the build is specific to my Nucleo H743ZI2 board but I think can be generalized for many other STM32 boards.
Please file an issue if you would like to have a build added for your board.
#36 moved the audio_frontend from a seperate module into a type within the microlite module.
The steps to build are self documented within the github actions used to build the firmware for the various supported boards. Look with the .github/workflows/ directory to see the pipeline scripts for each board.
Issues are welcomed to request adding ci support for new boards.
Follow the Upgrade Instructions on how to upgrade. The main issue is to get the 3 git submodules updated to the latest values.
Follow the Linux Build Instructions on how to build the latest firmware from a fresh clone of this repository.
The goal of this project is for experimenting with TinyML and the plan is to have micropython implementations of the examples from the tensorflow micro project.
In all cases the model.tflite can be used from upstream. However its common for us to have different implementation code written in micropython instead of C++.
Pull requests are welcome for additional examples with custom models.
| TF Micro Example Reference Examples | Training |
|---|---|
| hello_world | Train Hello World |
| magic_wand | Train Magic Wand |
| micro_speech | Train Micro Speech |
| person_detection | Train Person Detection |
Give a model an x value and it will give a y value. The chart of such points is an approximate sine wave:
Status:
- Works on unix port and esp32 port.
Process:
- Sample Audio
- Convert to spectrogram
- Set 1960 bytes on input tensor corresponding to 1 second worth of spectrogram.
- Run inference on model 3-4 times per second and average scores.
- Scores over 200 indicate a match
Status:
- Works on unix port with files.
- Works on esp32 no-spi ram board with machine.I2S.
ESP32 Example with INMP441 Microphone
Process:
- Capture Images
- Convert to 96x96 pixel int8 greyscale images
- Set on input later of model
- Run inference on image
- if person > no person it thinks the image is a person
- if no person > person it thinks the image contains no person
Status:
- Works on unix port and esp32 port using files.
Person Detection Documentation
At the moment we are using the main branch in the tensorflow lite micro repository.
This is the C++ api version of tensorflow lite designed to run on microcontrollers.
We are building from micropython master branch.
ESP32D0WDQ6 4MB Flash
Download the firmware from the latest ci build.
The zip file contains:
- The bootloader
- The partition table
- The firmware
esptool.py -p COM5 -b 460800 --before default_reset --after hard_reset --chip esp32 write_flash --flash_mode dio --flash_size detect --flash_freq 40m 0x1000 bootloader/bootloader.bin 0x8000 partition_table/partition-table.bin 0x10000 micropython.bin
TODO
The Micropython I2S implementation was written by Mike Teachman and its because of his hard work that the micro-speech example works so well.
As far as I am aware OpenMV (https://openmv.io/) was the first micropython firmware to support tensorflow.
I copied extensively from their approach to get inference working in the hello world example and also for micro-speech example.
I started from their libtf code for how to interact with the Tensorflow C++ API from micropython:
https://github.com/openmv/tensorflow-lib/blob/343fe84c97f73d2fe17a0ed23540d06c782fafe7/libtf.cc and https://github.com/openmv/tensorflow-lib/blob/343fe84c97f73d2fe17a0ed23540d06c782fafe7/libtf.h
The audio-frontend module originated by looking at how openmv connected to the tensorflow microfrontend here: https://github.com/openmv/openmv/blob/3d9929eeae563c5b370ac86afa9216df50f0c079/src/omv/ports/stm32/modules/py_micro_speech.c