/Lego51515BLEArduinoTensorFlow

Primary LanguagePythonGNU General Public License v3.0GPL-3.0

Lego51515BLEArduinoTensorFlow

Objective

The aim of this project is to create a Lego Mindstorms 51515 robot that can detect, locate and then pick up a pre-defined object using computer vision.

At the time of writing (Feb 2021) there are no 'off the shelf' official or third party sensors (that I am aware of!) that deliver an object detection capability for the Lego Mindstorms 51515 set. An object detection camera capability is available for the EV3 system by the third party Pixy2 camera. In fact, this EV3 solution using the Pixy camera was the inspiration for the current project:

In searching for a solution to this problem I established the following criteria:

  • Demonstrate the integration of computer vision into a Lego 51515 project
  • Demonstrate Bluetooth communication with a non-Lego peripheral (in this case an HM-10 module)
  • Create a project that can be relatively simply copied and extended by other users
  • For the Lego part of the solution, only use components from the stock 51515 set

Solution

I settled on a three part solution. Note that only the code elements are contained in this repository as the basic robot design is based on the Tricky robot in the 51515 set and the link to the grabber design is included below. The following gives an overview of the solution elements and design choices:

Part 1: Lego Mindstorms 51515 Robot

  • A single Lego 51515 set in stock configuration but loaded with the Spike Prime firmware. This is because as at Feb 2021, the Lego 51515 firmware does not include the ubluetooth module required for BluetoothLE comms
  • The robot design was based on the 'Tricky' robot from the 51515 set but with the addition of a grabber using this design. This grabber design has the benefit of grabbing and then lifting the object using a single motor
  • I then added some additional elements in order to create a deck for the Arduino board and battery etc.
  • The software consists of a single MicroPython file that contains code to a) connect to the HM-10 BluetoothLE module on the Arduino board and b) handle movement, edge detection, controlling the grabber etc.

Part 2: Arduino board

  • An Arduino board based on the Arduino Micro. The board also contains both an HC-05 Bluetooth Classic module and an HM-10 BluetoothLE module
  • The reason for using both types of Bluetooth module is that the object detection app described in the next section communicates via Bluetooth Classic, yet the Lego hub communicates via BluetoothLE

Part 3: Ard Object Tracker Android App

  • The final element in the solution is the Ard Object Detection app which can be found here (Android app)
  • The Ard Object Detection app provides an object detection capability via TensorFlow Lite and then exposes this as text over a Bluetooth Classic connection
  • This is the key element in the solution as it provides a way to detect objects without having to custom code a TensorFlow app in either Android or RPi Python
  • I did also consider using AppInventor, as this contains a number of object detection extensions, but unfortunately these solutions were limited or too slow
  • I also experimented with a strategy of sticking simple text to the objects to be picked up and programmatically taking pics via AppInventor and then passing these to the Google Cloud Text Detection API. This actually worked quite well but was too slow vs the Ard Detection App
  • Note that the Ard Object Detection app implements the COCO dataset. Therefore to avoid having the train a custom model, I decided to use a 'cup' as the object for the robot to pick up. Being a cyclinder, a cup has the added benefit that the robot grabber tool can pick it up more consistently regardless of approach angle

Key Challenges and Learnings

BluetoothLE

Using BluetoothLE on the Lego Prime/51515 hub is currently not that well supported and there are very few forum posts on the subject. I started to make much more progess however by reading the following:

Even with the above resources, there are a lot of device specific implementation issues. For instance:

  • I needed to use 16bit UUIDs to identify the services and characteristics of the HM-10 Bluetooth module. You will see a lot of code examples that use 128bit
  • The HM-10 doesn't seem to supply service data in a passive Bluetooth scan so I had to identify the device by the MAC address (contrary to most examples)
  • I had a lot of issues testing the Bluetooth features on the Lego hub and it took me ages to realise that for some reason you need to completely power down the hub and power up again between code executions. User Vinz alludes to this issue in the Eurobricks thread above. His solution was to detach the USB cable from the Lego hub. I didn't try that myself as just using the hub display for debug would be too limiting for me
  • I would strongly recommend reading an article like this to get an overview of how the BluetoothLE system works before delving into any code

Arduino

I used the Arduino Micro which has one hardware UART in addition to the one for the USB connector. This meant that I needed to put one of the Bluetooth adaptors on a SoftwareSerial connection. Let me save you a lot of time if you are going to use a different Arduino by sharing these links:

Other than that, I would say that the Arduino part of the solution was one of the simplest parts. The Arduino ecosystem is very well supported and coding is very immediate. In fact I decided to use Arduino rather than RPi for this element of the solution to avoid all the issues with Python setup on a RPi and the fact that there are multiple RPi Bluetooth libraries and they don't seem that well covered on forums. The benefit of the HC-05 and HM-10 modules on Arduino is that they use a simple UART serial implementation which (for me at any rate) simplified things enormously.

One thing to look out for when using the HC-05 and HM-10 modules is that internally they operate at 3.3v vs 5v used by most Arduinos. This necessitates a voltage divider to be implemented between the Tx of the Arduino and the Rx of the Bluetooth module. Here are a couple of useful articles on the HC-05 and HM-10 modules:

Here are some other helpful articles I found on my journey:

Spike Prime Python
BluetoothLE
Ard Object Tracking App
TensorFlow
AppInventor
RPi/Jetson Nano
Arduino