μbit Gesture Mapper

Requirements

NodeJS v14+
Python 3.8 (Python 3.9+ is currently not supported by TensorFlow)

Installing

Client

cd client
npm i

Server

It is recommended to use a python virtual environment or similar to avoid all the dependencies being added system wide to your python install.

To do this on windows:

cd server
python -m venv venv
venv\Scripts\activate

Note the script to activate the venv may be slightly different on other platforms.

Then installing the dependencies once the virtual environment is active:

pip install -r requirements.txt

If you need to exit the virtual environment run:

venv\Scripts\deactivate

from the server directory.

Running

To run both the client and server respectively it is:

cd client
npm start

cd server
flask run

If your local machine has Bluetooth connectivity, you can connect to the app over localhost and do not need to use HTTPS/Ngrok. For example, if the client and sever are running on a laptop that has a Bluetooth adapter, you can connect the micro:bit to the app using the URL http://localhost:3000. If this is not possible, you must setup HTTPS or use Ngrok as described below.

Ngrok

If you need to connect to a micro:bit to any device other than localhost, both the server and client need to be running behind Ngrok. Also the Ngrok URL for the server needs to be set in the configuration file found in client/src/config.json

Install Ngrok:

npm install --global ngrok

You may need to restart the terminal at this point if the ngrok command is not available in the system path.

Server:

ngrok http -region=eu 5000

Don't forget to update the server URL in client/src/config.json.

Client:

ngrok http -region=eu 3000

Note: default ports of server and client are 5000 and 3000 respectively.

Connecting the micro:bit to a desktop/laptop

You may have to enable a flag in the browser to allow Web Bluetooth connections on desktop. To do this go to the flags page, enable the Experimental Web Platform Features flag and re-launch the browser.

Browser Flags (Experiments) Page Locations

Enter the URL below in the address bar of your preferred browser to access the flags (Experiments) page

Google Chrome/Chromium: chrome://flags/#enable-experimental-web-platform-features
Microsoft Edge: edge://flags/#enable-experimental-web-platform-features

Issues

Bluetooth Connectivity

There seems to be several issues with bluetooth connectivity which can prevent the Microbit from connecting properly. The suboptimal workaround is to simply reconnect until the client can properly read the sensor data. Usually a reset on the Microbit helps here.

There is also other issues in these scenarios:

Using a battery pack: The Microbit tends to disconnect after a period of time.
Using bluetooth 4.0: The Microbit tends to disconnect after a period of time.

The solution to both of the above issues is to use less of the bluetooth services. This means that removing the Magnetometer service from the Microbit may help. Another untested solution is to lower the tick rate of the sensor readings so that there is less information to send.

TensorFlow on the Client

This issue can be described as the time taken for a forward pass through the machine learning model is greater than the speed at which we are sending in new data for the model to process. This means that there is a huge backlog of data and a delay in the readings. At one point, depending on your machines resources, the client will also crash due to this. Altering the tick rate on the sensors may also improve the current model however in addition another model suggestion is made below:

RNN

A suggested solution for future development is to use a Recurrent Neural Network. This allows us to work with time-series data rather than a static model. What this means is that instead of the current approach of sending all 30 ticks every new tick to the model, we will only need to send one every tick. This will reduce the amount of processing recquired by a large degree.

Web Audio

For details on the Web Audio implementation in this project, see docs/audio.md

Contributions and Thanks

This project was developed by Internet of Things Applications class 2021 taught by Jason Berry.

Based and influenced by code from https://github.com/WIT-IoT-Apps-2021/microbit-ble-vue

Web audio sound effect nodes based on https://github.com/cwilso/Audio-Input-Effects

All Contributors (emoji key):

_{Max Larkin} 💻 🤔	_{Dylan Gore} 💻 🤔	_{Jack Power} 🤔	_{Adam Cotter} 🤔	_{Robert Solomon} 🤔
_{Jun-Shuo Ng} 🤔	_{Ashraf Mustafa} 🤔	_{Ewan Hills} 🤔