SensorsIot/IOTstack

Experiments with Bluetooth (exploration prior to documentation)

Closed this issue · 3 comments

Paraphraser commented

IOTstack – Experiments with Bluetooth

This issue documents my approach to getting node-red-contrib-generic-ble working under IOTstack. I'm hoping that people with a lot more experience than me will spot any flaws in my approach.

I'm not going to keep typing node-red-contrib-generic-ble because it's boring. I'm going to use the abbreviation BLE.

I don't have a use-case for Bluetooth so I can't go any further than I've documented here. It would be very helpful to get confirmation that the result is fit for purpose and that useful communications can be achieved between Node-Red and Bluetooth devices.

If there's enough interest, I will volunteer to turn this into a documentation page for the IOTstack project. If someone else wants to tackle the documentation using this issue as a starting point, please go right ahead.

Installation

Give the current user access to the Bluetooth hardware

$ sudo usermod -G bluetooth -a $(whoami)
$ sudo reboot

Update to the latest and greatest of everything

Make sure Raspbian (ahem, "Raspberry Pi OS") is up-to-date:

$ cd ~/IOTstack
$ docker-compose down
$ cd
$ sudo apt update
$ sudo apt upgrade
$ sudo reboot

Make sure IOTstack is up-to-date:

$ cd ~/IOTstack
$ git pull origin master

Make sure container images are up-to-date:

$ docker rmi "iotstack_nodered" "nodered/node-red"
$ docker-compose pull
$ docker-compose up --build -d

Demonstrate that nodered is in non-host mode (the default)

In the following, 192.168.132.1 is my local router.

$ docker exec nodered traceroute 192.168.132.1 
traceroute to 192.168.132.1 (192.168.132.1), 30 hops max, 38 byte packets
 1  172.18.0.1 (172.18.0.1)  0.011 ms  0.015 ms  0.012 ms
 2  myrouter.mydomain.com (192.168.132.1)  0.288 ms  0.263 ms  0.143 ms
$

Two hops = non-host mode.

Edit the nodered configuration

Edit ~/IOTstack/docker-compose.yml and add these lines:

      - /var/run/docker.sock:/var/run/docker.sock
      - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
    devices:
      - "/dev/ttyAMA0:/dev/ttyAMA0"
      - "/dev/vcio:/dev/vcio"
      - "/dev/gpiomem:/dev/gpiomem"
    network_mode: "host"

The final result should be:

  nodered:
    container_name: nodered
    build: ./services/nodered/.
    restart: unless-stopped
    user: "0"
    privileged: true
    env_file: ./services/nodered/nodered.env
    ports:
      - 1880:1880
    volumes:
      - ./volumes/nodered/data:/data
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
    devices:
      - "/dev/ttyAMA0:/dev/ttyAMA0"
      - "/dev/vcio:/dev/vcio"
      - "/dev/gpiomem:/dev/gpiomem"
    network_mode: "host"

Remember, you can't use tabs in docker-compose.yml. You must use spaces.

Apply the changes:

$ docker-compose up -d
…
Recreating nodered ... done
$

Confirm that nodered is in host mode

$ docker exec nodered traceroute 192.168.132.1
traceroute to 192.168.132.1 (192.168.132.1), 30 hops max, 38 byte packets
 1  myrouter.mydomain.com (192.168.132.1)  0.252 ms  1.610 ms  0.200 ms
$

One hop = host mode.

Confirm that Node-Red (the service) is reachable

You can either open a browser and point it to port 1880 on your Raspberry Pi's IP address or domain name, or you can do it from the command line:

$ curl -I localhost:1880
HTTP/1.1 200 OK
…
$

As long as you get the "200 OK" you should be fine.

Install node-red-contrib-generic-ble

Open a shell into the nodered container:

$ docker exec -it nodered bash
#

Confirm that BLE is not already installed:

# ls -ld node_modules/node-red-contrib-generic-ble
ls: node_modules/node-red-contrib-generic-ble: No such file or directory
#

BLE has a dependency that the package does not take care of:

# apk add eudev-dev
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/armv7/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/armv7/APKINDEX.tar.gz
(1/2) Installing pkgconf (1.6.3-r0)
(2/2) Installing eudev-dev (3.2.9-r1)
Executing busybox-1.31.1-r9.trigger
OK: 240 MiB in 67 packages
#

Install BLE:

# npm install node-red-contrib-generic-ble

The compile chucks out a lot of notes and warnings but succeeds.

Confirm that BLE has been installed:

# ls -ld node_modules/node-red-contrib-generic-ble
drwxr-xr-x    5 root     root          4096 Jun  3 21:00 node_modules/node-red-contrib-generic-ble
#

Exit the container shell:

# exit
$

Restart nodered. This picks up the new BLE node:

$ docker-compose restart nodered

Check for errors:

$ docker logs nodered

The output I get is:

> node-red-docker@1.0.6 start /usr/src/node-red
> node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data"

3 Jun 21:05:05 - [info] 

Welcome to Node-RED
===================

3 Jun 21:05:05 - [info] Node-RED version: v1.0.6
3 Jun 21:05:05 - [info] Node.js  version: v10.20.1
3 Jun 21:05:05 - [info] Linux 4.19.118-v7l+ arm LE
3 Jun 21:05:06 - [info] Loading palette nodes
3 Jun 21:05:08 - [info] Dashboard version 2.22.1 started at /ui
3 Jun 21:05:09 - [info] Settings file  : /data/settings.js
3 Jun 21:05:09 - [info] Context store  : 'default' [module=memory]
3 Jun 21:05:09 - [info] User directory : /data
3 Jun 21:05:09 - [warn] Projects disabled : editorTheme.projects.enabled=false
3 Jun 21:05:09 - [info] Flows file     : /data/flows.json
3 Jun 21:05:09 - [info] Server now running at http://127.0.0.1:1880/
3 Jun 21:05:09 - [warn] 

---------------------------------------------------------------------
Your flow credentials file is encrypted using a system-generated key.

If the system-generated key is lost for any reason, your credentials
file will not be recoverable, you will have to delete it and re-enter
your credentials.

You should set your own key using the 'credentialSecret' option in
your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.
---------------------------------------------------------------------

3 Jun 21:05:09 - [info] Starting flows
3 Jun 21:05:09 - [info] Started flows
3 Jun 21:05:09 - [info] [sqlitedb:eb5d5c76.848308] opened /data/Databases/Mailbox/mailbox.db ok
3 Jun 21:05:09 - [info] [sqlitedb:f51495af.860dd8] opened /data/Databases/TrigBoard/trigboard.db ok
3 Jun 21:05:09 - [info] [mqtt-broker:Merle MQTT] Connected to broker: mqtt://mosquitto:1883
3 Jun 21:05:09 - [info] [mqtt-broker:Merle MQTT] Connected to broker: mqtt://mosquitto:1883
3 Jun 21:05:09 - [info] [mqtt-broker:Merle MQTT] Connected to broker: mqtt://mosquitto:1883
3 Jun 21:05:09 - [info] [mqtt-broker:Merle MQTT] Connected to broker: mqtt://mosquitto:1883
3 Jun 21:05:09 - [info] [mqtt-broker:Docker MQTT] Connected to broker: mqtt://mosquitto:1883
$

No errors!

Testing

  1. Use your browser to connect to Node-Red.
  2. Created a new empty flow.
  3. Scroll down in the Node Palette until you see "Generic BLE in" and drag it onto the canvas.
  4. Double-click the BLE-in node to open its properties.
  5. Click the pencil icon next to "Add new Generic BLE...".
  6. Turn on the "BLE Scanning" checkbox.
  7. Wait. If all goes well, the "Scan Result" popup menu will populate and you will be able to open that popup menu and see all your Bluetooth MACs.

BLEWorking

Discussion

First attempt at getting BLE working (failure):

  • Starting docker-compose.yml configuration:

       nodered:
     container_name: nodered
     build: ./services/nodered/.
     restart: unless-stopped
     user: "0"
     privileged: true
     env_file: ./services/nodered/nodered.env
     ports:
       - 1880:1880
     volumes:
       - ./volumes/nodered/data:/data

  • I forced a complete rebuild of iotstack_nodered. The nodes installed at build time were the defaults plus SQLite.

  • I edited docker-compose.yml, to add:

         network_mode: "host"

    Then restarted nodered and checked that it was in host mode.

  • At HilscherAutomation I came across the requirement for:

         devices:
       - "/dev/ttyAMA0:/dev/ttyAMA0"
       - "/dev/vcio:/dev/vcio"
       - "/dev/gpiomem:/dev/gpiomem"

    There were more lines in the HilscherAutomation list but the above three were the only ones that mapped to anything on my RPi4. They seemed like a good idea but I do not know whether they are required.

  • My first attempt at compiling BLE failed because of a missing dependency. I worked out how to install eudev-dev, after which BLE compiled.

  • Restarting the nodered container to activate BLE produced an error about /var/run/docker.sock so I added a mapping:

         volumes:
       - ./volumes/nodered/data:/data
       - /var/run/docker.sock:/var/run/docker.sock

  • That cured the docker.sock problem but nodered would still not come up. The new error was:

     Error: connect ENOENT /var/run/dbus/system_bus_socket at PipeConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)

  • I tried to solve that by adding another mapping:

         volumes:
       - ./volumes/nodered/data:/data
       - /var/run/docker.sock:/var/run/docker.sock
       - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket

    but nodered kept restarting with the same error.

I checked a few more things (eg possible permission conflicts) but that's where I gave up.

Second attempt at getting BLE working (success):

  • Starting docker-compose.yml configuration:

       nodered:
     container_name: nodered
     build: ./services/nodered/.
     restart: unless-stopped
     user: "0"
     privileged: true
     env_file: ./services/nodered/nodered.env
     ports:
       - 1880:1880
     volumes:
       - ./volumes/nodered/data:/data
       - /var/run/docker.sock:/var/run/docker.sock
       - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
     devices:
       - "/dev/ttyAMA0:/dev/ttyAMA0"
       - "/dev/vcio:/dev/vcio"
       - "/dev/gpiomem:/dev/gpiomem"

  • I forced a complete rebuild of iotstack_nodered. The nodes installed at build time were the defaults plus SQLite.

  • I edited docker-compose.yml, to add:

         network_mode: "host"

    Then restarted nodered and checked that it was in host mode.

  • I installed the eudev-dev dependency and compiled BLE.

  • I restart the nodered container.

And it just worked.

To say that I was astonished would be a severe understatement.

My guess (and it is no more than a guess) is that some or all of the volumes and/or devices additions to docker-compose.yml needed to be in place before BLE was compiled.

More work

I tried to see if I could add node-red-contrib-generic-ble to the menu but that, of course, came with the eudev-dev pre-requisite. I tried adding:

RUN apk add eudev-dev

as the second line of the Dockerfile but docker-compose hated it.

I will leave it to others to solve this problem.

Just be aware that, in the meantime, a rebuild of nodered will require a manual install of BLE.

Paraphraser commented

I solved the problem described under "More work". This is the Dockerfile I used:

FROM nodered/node-red:latest
USER root
RUN apk update
RUN apk upgrade
RUN apk add --no-cache eudev-dev
RUN for addonnodes in \
node-red-node-pi-gpiod \
node-red-dashboard \
node-red-contrib-influxdb \
node-red-contrib-generic-ble \
; do \
npm install ${addonnodes} ;\
done;
RUN npm install --unsafe-perm node-red-node-sqlite

Build process:

$ cd ~/IOTstack
$ docker-compose down
$ docker rmi "iotstack_nodered"
$ docker-compose up --build -d

Notes:

  1. The permissions problem that I hit before was solved by adding USER root.

  2. The base nodered/node-red image is built on Alpine Linux, which is why apk is needed instead of apt.

  3. These three lines could probably be hard-coded:

    USER root
RUN apk update
RUN apk upgrade

  4. The RUN apk add --no-cache eudev-dev could also be hard-coded. It's only providing a library. Other than occupying some compile time, it would do no harm even if BLE was not being installed. The alternative would be to adopt the pattern for SQLite and use a flag to turn installation on when appropriate.

  5. The --no-cache flag seems to appear in most examples of using apk to install packages via Dockerfiles. I tried building both with and without that option. It worked either way so I can't say whether it is required.

  6. node-red-contrib-generic-ble works OK as an argument to the for-loop (ie it doesn't need special handling like node-red-node-sqlite).

Paraphraser commented

Revised Magic Incantation

More research has led me to conclude that "host mode" is not required. The Docker documentation says:

Network Description
host Use the host's network stack inside the container

In my testing, I can see nearby Bluetooth MACs via a "Generic BLE in" node irrespective of whether "host" mode is engaged or not.

Permissions

$ sudo usermod -G bluetooth -a $(whoami)
$ sudo reboot

docker-compose.yml

  nodered:
    container_name: nodered
    build: ./services/nodered/.
    restart: unless-stopped
    user: "0"
    privileged: true
    env_file: ./services/nodered/nodered.env
    ports:
      - 1880:1880
    volumes:
      - ./volumes/nodered/data:/data
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
    devices:
      - "/dev/ttyAMA0:/dev/ttyAMA0"
      - "/dev/vcio:/dev/vcio"
      - "/dev/gpiomem:/dev/gpiomem"

The change from the default is to add the last six lines.

Dockerfile

FROM nodered/node-red:latest
USER root
RUN apk update
RUN apk upgrade
RUN apk add --no-cache eudev-dev
RUN for addonnodes in \
node-red-node-pi-gpiod \
node-red-dashboard \
node-red-contrib-influxdb \
node-red-contrib-generic-ble \
; do \
npm install ${addonnodes} ;\
done;
RUN npm install --unsafe-perm node-red-node-sqlite

Build

$ cd ~/IOTstack
$ docker-compose down
$ docker rmi "iotstack_nodered"
$ docker-compose up --build -d

Test

  1. Use your browser to connect to Node-Red.
  2. Created a new empty flow.
  3. Scroll down in the Node Palette until you see "Generic BLE in" and drag it onto the canvas.
  4. Double-click the BLE-in node to open its properties.
  5. Click the pencil icon next to "Add new Generic BLE...".
  6. Turn on the "BLE Scanning" checkbox.
  7. Wait.

If all goes well, the "Scan Result" popup menu will populate and you will be able to open that popup menu and see all your Bluetooth MACs.

Paraphraser commented

Prepared pull request #70