In this project I use the newly added TensorFlow Lite support for 32 bit Arduino boards (not all Arduino boards are 32 bit, in fact, most of them are 8 bit).
I made a quick neural network that recognised several fighting movements such as uppercut, punch, slash and stabbing using the Arduino Nano 33 BLE Sense which already includes a gyro and accelerometer sensor.
In this youtube video we can see it working in real time.
This project takes inspiration from the one found in the Arduino Blog about training your own Gesture Recognition Classifier
Each movement was first captured 10 times using the first script IMU_Capture with the Arduino IDE and then saving that gyro and accelerometer data into different .csv file.
Then the Neural Network was trained on my desktop computer using the Tensorflow_to_TensorflowLite_Converter which uses the .csv files we obtained before, as training examples for the Neural Network. When we have trained the Network, we convert it to a model.h file that can be compiled for an Arduino or any other Microcontroller, such as an ESP32.
Lastly we attach that model.h file into the IMU_Classifier which already has all the TensorFlow Lite instructions that will predict the movement.
Let's get a bit more in depth with each step of the process:
We will need to have a 3.5.x to 3.7.x version of Python installed on our computer (Those are the only Python versions with TensorFlow 2.0 support by now). We will also need to have a 2.x version of TensorFlow in our computer, we can install it using the following command in our console or virtual enviorenment if we are using one:
pip3 install tensorflow
We will also be using a Jupyter Notebook in our second step, therefore we also have to install an instance of it:
pip install jupyterlab
Since we are using a non standard Arduino board, the Arduino Nano 33 BLE Sense, we need to add it to the Board Manager in our Arduino IDE in Tools > Board > Boards Manager
We also have to add the library that reads the LSM9DS1 library in order to read from the IMU (Inertial Measurement Unit) inside Tools > Manage Libraries
Then we can plug the Arduino Nano 33 to our computer's USB, choose the right board in Tools > Board > Arduino Nano 33 BLE and choose the right port.
Now we are ready to get into the first script.
This short script simply starts recording all the gyro and accelerometer values into a comma separated format once the device goes over a certain accelerationThreshold defined at the beginning. Several tests have determined that 1.55 is a good empirical value to start recording movements with a certain force.
Once the device has gone over that threshold it will start outputting gyro and accelerometer data at 100Hz for 119 samples in the Serial Port, found in Tools > Port > Arduino Nano 33 BLE. Then it will stop outputting samples into the serial port until another sudden movement goes over that threshold.
We will record 10 instances of each of the movements. We will copy what we get in the Serial port and copy it into different .csv files, one file for each movement. Those files have already been created and copied into the second part of this tutorial.
In this part we can either train online the Neural Netwrok using a Colab created by Sandeep Mistry or use the slightly modified version I provide and train it in our machine with Jupyter Notebook. The proccess is easily explained in both the Notebooks.
Although some of the highlights of the Notebook could be that we "One Hot Encode" the dataset, that means labelling the data like this
| Data | Punch | Upper |
|---|---|---|
| 0,424 0,534... | 1 | 0 |
| 0,574 0,153... | 1 | 0 |
| 0,863 0,785... | 0 | 1 |
| 0,583 0,135... | 0 | 1 |
Instead of naming the target variable as target = ("Punch","Upper") and then encoding the data like this
| Data | Target |
|---|---|
| 0,424 0,534... | 1 |
| 0,574 0,153... | 1 |
| 0,863 0,785... | 2 |
| 0,583 0,135... | 2 |
Because the Neural Network could interpret that a number in the middle, like 1.5, makes sense, therefore turning a categorical variable, like Punch, Upper... into a numerical variable, like height or stock value. So our error function the Mean Squared Error takes that into account when we hot encode the data.
We could also use another function like the Sparse Categorical Crossentropy which already takes into account a variable with different categorical values.
We also have to define a Neural Network Architecture. This time we used a simple Neural Network with two Dense Layers-
Lastly we simply put the model.h we obtained in the last step in our working folder. Compile the .ino program and upload it into the Arduino and the program is ready to go. Youtube video
Some considerations to take into account are:
-
The Neural Network is very sensitive to the position, inclination, velocity and acceleration of the movements you record. So try to keep the Arduino in the same spot during the recording and later during the predictions
-
TensorFlow Lite in microcontrollers can deal with much more complex Neural Networks, like several layers deep and complex layers such as convolutional ones. But keep in mind that bigger isn't always better. As there are two main problems with this approach:
- Training will be slower, and the Neural Network might not find a local minima and never converge. Outputting worse than random results
- Compiling will take longer and take up more space in the microcontroller.
-
Even though processing a Neural Network is very resource heavy on the microcontroller. The measured current draw between the microcontroller on idle (17 mA) and using a Neural Net (21 mA) was a 20% difference, although the device I used for measuring has a precission of +- 2 mA, so results may not be all that accurate.