Node-RED MCU Edition

Introduction

This document introduces an implementation of the Node-RED runtime that runs on resource-constrained microcontrollers (MCUs). Node-RED is a popular visual environment that describes itself as "a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways."

Node-RED is built on Node.js and, consequently, runs where Node.js does: desktop computers and single-board computers lfike the Raspberry Pi. Because of the dependency on Node.js, Node-RED cannot run where Node cannot, notably the low-cost MCUs found in many IoT products and popular in the maker community.

These MCUs are able to run the same JavaScript language used by Node-RED thanks to the XS JavaScript engine in the Moddable SDK. However, these MCUs have much less RAM, much less CPU power, and an RTOS instead of a Linux-based OS. As a result, they require a very different implementation. A typical target microcontroller is the ESP32, running FreeRTOS with about 280 KB of free RAM and a 160 MHz CPU clock.

The implementation converts the JSON descriptions output by Node-RED to JavaScript objects that are compatible with MCUs. The implementation uses standard JavaScript running in the Moddable SDK runtime. The ECMA-419 standard, the ECMAScript® Embedded Systems API Specification, is used for I/O such as access to pin hardware and networking.

This effort is intended to evaluate whether it is feasible to support Node-RED on MCUs. To achieve that goal, the focus is on a breadth rather than depth. Node-RED has a wealth of features that will take considerable time to implement well. This proof-of-concept effort has the following characteristics:

Implements fundamental structures
Supports core nodes that help to explore overall architecture
Requires no changes to Node-RED
Not overly concerned with efficiency

Efficiency is essential for resource constrained device, but it is not a priority yet. For the current goal of determining if it is realistic to support Node-RED on MCUs, the projects just need to run, not run optimally.

Method

The conversion from Node-RED JSON objects to JavaScript objects happens in two separate phase. The first phase happens during the process of building a Node-RED MCU project; the second phase, while running the project.

The following sections look at some examples of the transform, starting from the Node-RED configuration of each object.

HTTP Request

Here is a typical HTTP Request node.

Node-RED generates this JSON for the HTTP Request.

{
    "id": "b0bc5df11987f5b1",
    "type": "http request",
    "z": "8f44c46fbe03a48d",
    "name": "JSON Request",
    "method": "GET",
    "ret": "obj",
    "paytoqs": "ignore",
    "url": "",
    "tls": "",
    "persist": false,
    "proxy": "",
    "authType": "",
    "senderr": true,
    "credentials": {},
    "x": 420,
    "y": 420,
    "wires": [
        [
            "601862f19774933c"
        ]
    ]
}

The nodered2mcu tool converts this JSON to two JavaScript calls. The first creates the node and adds it to the flow. The second initializes the node with a subset of the properties of the Node-RED generated JSON:

createNode("http request", "b0bc5df11987f5b1", "JSON Request", flow);

node.onStart({
	method: "GET",
	ret: "obj",
	paytoqs: "ignore",
	url: "",
	tls: "",
	persist: false,
	proxy: "",
	authType: "",
	senderr: true,
	credentials: {},
	wires: [["601862f19774933c"]],
});

This JavaScript is compiled into the Node-RED MCU Edition project and run on the device. The implementation of the HTTPRequestNode uses this JSON to implement the HTTP request. In this case, the URL for the HTTP request is provided by an Inject node. The next section shows the implementation of the inject.

Inject

This Inject node is configured to send a URL to the HTTP Request node that its output is connected to.

Node-RED generates this JSON for the Inject node.

{
    "id": "b0afa70581ce895a",
    "type": "inject",
    "z": "8f44c46fbe03a48d",
    "name": "GET httpbin/json",
    "props": [
        {
            "p": "url",
            "v": "httpbin.org/json",
            "vt": "str"
        }
    ],
    "repeat": "",
    "crontab": "",
    "once": true,
    "onceDelay": "6",
    "topic": "",
    "x": 190,
    "y": 420,
    "wires": [
        [
            "b0bc5df11987f5b1"
        ]
    ]
}

The nodered2mcu tool is able to optimize this JSON considerably when generating the two JavaScript calls.

createNode("inject", "b0afa70581ce895a", "GET httpbin/json", flow);

node = nodes.next().value;	// inject - b0afa70581ce895a
node.onStart({
	wires: [["b0bc5df11987f5b1"]],
	trigger: function () {
		const msg = {};
		msg.url = "httpbin.org/json";
		this.send(msg);
	},
	initialize: function () {
		Timer.set(() => this.trigger(), 6000);
	},
});

By converting to JavaScript, nodered2mcu can include JavaScript functions in addition to JSON values. It creates a trigger function to perform the inject operation described by the JSON. It creates an initialize function to set up a timer that invokes trigger after six seconds. This optimization allows the Inject node to more quickly than interpreting the JSON configuration at runtime in Node-RED MCU Edition. It also tends to require less RAM.

Change

Here is a typical Change node.

Node-RED generates this JSON for the HTTP Request.

{
    "id": "734223794d10f7cc",
    "type": "change",
    "z": "8f44c46fbe03a48d",
    "name": "",
    "rules": [
        {
            "t": "move",
            "p": "payload",
            "pt": "msg",
            "to": "DEBUGGER",
            "tot": "msg"
        },
        {
            "t": "set",
            "p": "one",
            "pt": "flow",
            "to": "one one one one",
            "tot": "str"
        }
    ],
    "action": "",
    "property": "",
    "from": "",
    "to": "",
    "reg": false,
    "x": 680,
    "y": 320,
    "wires": [
        [
            "e34b347db64fcdc8"
        ]
    ]
}

The nodered2mcu tool is able to reduce this large block of JSON to a JavaScript function of just a few lines. The onMessage function it generates is the implementation of the Change node's message handler, so there is no additional overhead in processing this node.

createNode("change", "734223794d10f7cc", "", flow);

node.onStart({
	wires: [["e34b347db64fcdc8"]],
	onMessage: function (msg) {
		msg.DEBUGGER = msg.payload;
		delete msg.payload
		this.flow.set("one", "one one one one");
		return msg;
	},
});

Results

This initial effort successfully runs useful, if small, Node-RED projects on MCUs. They run reliably and perform well. They have been tested on ESP32 and ESP8266 MCUs. The ESP8266 is quite constrained, running at 80 MHz with only about 45 KB of RAM. Both are able to run flows that connect physical buttons and LEDs to the cloud using MQTT. The following sections provide details on what has been implemented.

This effort was helped greatly by Node-RED's small, well-designed core architecture. That simplicity minimizes what needed to be implemented to execute the nodes and flows. Node-RED achieves its richness through the addition of nodes on top of its core architecture. This minimal and modular approach is well suited to MCUs which are designed to be capable enough to do useful work, not to be a scalable, general-purpose computing device.

This effort was also made easier by both Node-RED and the Moddable SDK being built on JavaScript engines that provide standard, modern JavaScript (V8 and XS, respectively). This is essential because the Node-RED runtime is itself implemented in JavaScript and makes JavaScript accessible to developers through the Function node. Additionally, the capabilities defined by ECMA-419 are often a direct match for the runtime needs of Node-RED, making implementation of embedded I/O capabilities remarkably straightforward.

Based on these early results, it seems possible to provide a valuable implementation of Node-RED for MCUs. This would make developing software for these devices accessible to more developers thanks to Node-RED's goal of providing "low-code programming." It would also allow developers with Node-RED experience a path to extend their reach to widely-used MCUs.

Ways to Help

Based on this investigation, bringing Node-RED to MCUs seems both possible and desirable. It is not a small effort, however. It will require expertise in embedded software development, ECMA-419, the Moddable SDK, the Node-RED runtime, and developing flows with Node-RED. Here are some ways you might help:

Help spread the world.
Give it a try. Share what you find – what works and what doesn't.
Help implement some of the incomplete features of the initial node suite
Implement additional notes
Help move the Node-RED Embedded Edition APIs to better match those in Node-RED
Implement Node-RED nodes for features of the hardware using ECMA-419
Implement Node-RED nodes for feature for the Moddable SDK
Help improve the documentation
Help improve the workflow for developers to deploy and debug flows

Please open an issue or submit a pull request on this repository on GitHub. You can also reach out to @phoddie and @moddabletech on Twitter or chat in real time on our Gitter page.

Running Flows on an MCU

Node-RED MCU Edition is a Moddable SDK project. It is built and run just like any Moddable SDK project. Flows run on ESP8266 and ESP32 MCUs, and in the Moddable SDK simulator on macOS computers. Node-RED MCU Edition requires Moddable SDK from August 8, 2022 (or later).

Of course, the Node-RED flows must be added to the project. The JSON version of the flows is stored in the nodes.json source file. There are two part to moving a Node-RED project to the Node-RED MCU Edition project.

The first is exporting the project from Node-RED.

Open the Node-RED project
From the Node-RED menu (top-left corner), select Export
On the Export tab, select "all flows"
Select Clipboard (not Local)
Select JSON (not Export nodes)
Select "Copy to Clipboard"

The JSON version of the flows is now on the clipboard. The second step is adding this JSON to the Moddable SDK project:

Open the flows.json file in the Node-RED MCU Edition project
Paste in the Node-RED JSON data

Build and run the Moddable SDK project as usual for the target device. The flows.json file is transformed by nodered2mcu as part of the build. If an error is detected, such as an unsupported feature, an error message is output and the build stops.

This process is quick and easy for early exploration. Of course, there are many ways it could be streamlined to improve the developer experience.

Structure

The Node-RED runtime executes the nodes and flows. This runtime architecture determines how nodes interact with each other. It also is a key factor in how efficiently the execution uses the limited RAM and CPU power available.

This is a summary of what is implemented in the Node-RED for MCUs runtime:

Nodes

This section lists the supported nodes. The implemented features are checked.

Comment

Supported

Debug

Console output is displayed in the xsbug log pane
Sidebar output is displayed in the xsbug message pane
Display of selected property or complete message
Output to node status

Function

"On Start" and "On Message" handlers
Access to Node context, flow context, and global context
Report uncaught exceptions to Catch nodes
Import modules (Setup)
When "On Start" returns Promise, queue received messages until ready
Does not wrap setTimeout (and friends) to automatically clear on stop
env(() to access flow's environment variables

Inject

Injects multiple properties
Property values Boolean, timestamp, JSON, number, string, and buffer
"Inject once after"
"Repeat after"
Property values msg., flow., global.
Interval between times
At a specific time
Property values expression & environment variable

Link Call

Implemented
Timeout

Link In

Implemented

Link Out

"Send to all connected link nodes"
"Return to calling link node"

Catch

"Catch errors from all nodes"
"Catch errors from selected nodes"
"Ignore errors handled by other Catch nodes"

Status

"Report status from all nodes"
"Report status from selected nodes"

GPIO In

Implemented using "rpi-gpio in" node

Select GPIO pin
Pull-up and pull-down resistor options
Debounce
Read initial state of pin on deploy/restart

GPIO Out

Select GPIO pin
Digital and PWM output modes
Initialize initial pin state option

Implemented using "rpi-gpio out" node with ECMA-419 Digital and PWM classes.

DS18B20

Multiple temperature sensors
Individual messages or array
Use msg.array to select output format
Use topic to select single sensor
id property in output matches Node-RED

Implemented using "rpi-ds18b20" node with OneWire bus module and DS18X20 temperature sensor module. Uses simulated temperature sensors on platforms without OneWire support.

MQTT Broker

Implemented using ECMA-419 MQTT Client draft.

MQTT In

Subscribe to topic with QoS 0
Payload formats: UTF-8, buffer, JSON, and Base64
Wildcards in topic
Dynamic subscription
Payload format: auto-detect

MQTT Out

Data formats: number, string, object, buffer
Retain flag

HTTP Request

Implemented using fetch based on ECMA-419 HTTP Client draft.

WebSocket Client

Implemented using HTML5 WebSocket based on ECMA-419 WebSocket Client draft.

WebSocket In

"Connect to"
"Listen on"

WebSocket Out

"Connect to"
"Listen on"

Range

Change

Delete property
Move property
Set property value
Property values Boolean, timestamp, JSON, number, string, and buffer
msg., flow. and global. targets
Replace within property value
Property values expression & environment variable

Switch

Multiple rules
"checking all rules"
"stopping after first match"
==, !=, <, <=, >, >=, is between, is true, is false, is null, is not null, is of type, otherwise, has key
flow., global., expression
env variable
is of type: buffer
contains, matches regexp, is empty, is not empty, sequence rules, JSONata exp

Filter

Split

JSON

Convert between JSON String & Object, Always Convert to JSON String, Always Convert to JSON Object
Format JSON string

File Write

Filename from node or message
Encoding from node or message
Encodings: auto, UTF-8, binary, hex, Base64
Actions: append to file, overwrite file, delete file
Add newline to each payload
Create directory if doesn't exist

The File Write node is implemented using the Moddable SDK of integration LittleFS.

File Read

Filename from node or message
Encodings: UTF-8, binary, hex, Base64
Read full file
Stream one line or buffer at a time
Report errors to Catch nodes

The File Read node is implemented using the Moddable SDK of integration LittleFS.

MCU Sensor

Retrieve sensor samples from any ECMA-419 Sensor Class Pattern driver
Simulated implementation for testing in full Node-RED
Properties sheet for configuration in Node-RED editor

See the MCU Sensor module's documentation for further details.

Delay

Everything

Note: The Delay node is the full Node-RED implementation (with a small, optional change to reduce its RAM footprint). This is possible by using the Compatibility Node.

Compatibility Node

The Compatibility Node runs nodes written for Node-RED. It is able to run the lower-case example from "Creating your first node" without any changes.

The Compatibility Node is tricky for a number of reasons. At this time, it should be considered a proof-of-concept and a foundation for future work.

The Node-RED nodes, including the lower-case example, are written as CommonJS modules. The XS JavaScript engine supports only standard ECMAScript modules (ESM). This leads to some limitations: export.modules can only be set once and any other exported properties are ignored. Because all modules are loaded as standard ECMAScript modules, nodes run by the Compatibility Node run in strict mode.

config passed to node implementation for initialization
.on() and .off() to register event handlers
"input" and "close" events
node.send() and send() to send messages
.log(), .warn(), and .error()
.status()
done and .done()

While a degree of source code compatibility is provided, the Compatibility Node does not attempt to emulate the (substantial) Node.js runtime. Consequently, it only runs nodes compatible with the features available in the Moddable SDK runtime. Nodes must be added to the Node-RED manifest to be included in the build. See the lower-case example in this repository for an example.

Note: The CompatibilityNode is implemented as a subclass of Node, the fundamental node type of the Node-RED MCU Edition runtime. The CompatibilityClass adds features for compatibility that use more memory and CPU power. For efficiency, internal nodes (inject, split, http-request, etc.) are implemented as subclasses of Node.

Future Work

This prototype is a breadth-first effort to implement all the steps required to execute meaningful Node-RED flows on a resource-constrained microcontroller. For compatibility and completeness, a great deal of work remains. That work requires many different kinds of experience and expertise. Evolving this early proof-of-concept to a generally useful implementation will require contributions from many motivated individuals.

The compatibility goal should be to provide the same behaviors as much as possible so that existing Node-RED developers can apply their knowledge and experience to embedded MCUs without encountered confusing and unnecessary differences. The goal is not to provide all the features of Node-RED, as some are impractical or impossible on the target class of devices.

Transformation

In this prototype, the nodes and flows exported by Node-RED are converted from JSON to JavaScript instances on the embedded device. This is a relatively heavy operation, involving several passes over the project structure and evaluating JavaScript code. Almost all of this work could be done as a build step on the development computer prior to deploy. Done well, this would free up additional RAM for the project, allow projects to start running faster, and detect use of unsupported Node-RED features earlier. This work should be done sooner than later, as it will change the class definition for the nodes.

Create JavaScript tool to convert Node-RED JSON to JavaScript source code
Integrate embedded conversion into Deploy feature of Node-RED (excellent work is being done here by @@ralphwetzel with the node-red-mcu-plugin)

Runtime

Align runtime behavior and APIs with Node-RED as much as practical. This would benefit from assistance from developers familiar with Node-RED.
Should messages sent between nodes should be sent asynchronously to avoid JavaScript stack overflows on long chains of nodes (what does Node-RED do here?)
Implement support to instantiate nodes from a Mod. This would allow updated flows to be installed on embedded devices in seconds.

Nodes

Possible future work on built-in nodes:

Common nodes. The Complete node appears to require Node-RED runtime behaviors beyond what this exploration now implements. It should be implemented sooner to ensure that the object design can support all the fundamental behaviors required.
Function nodes. The Trigger nodes appear to be essential. For the most part they should be straightforward to implement, though some of the behaviors are non-trivial. Exec and Template may not make sense.
Network nodes. The TCP and UDP nodes should be possible to implement using ECMA-419 in the same way MQTT has been implemented. (Note: all network support is client. Server support is also possible, but not yet explored.)
Sequence nodes. The Join, Sort, and Batch nodes should be possible to support. Like the Function nodes, some are quite sophisticated.
Parser. CSV should be possible to support, but the others (HTML, YAML, XML) are likely impractical.
Storage Watch file may not be useful, since there are no other processes modifying files. At best, it could monitor for changes made by other nodes.

The built-in nodes are useful for compatibility with the standard Node-RED behaviors. Additional nodes should be added to support embedded features. For example, a Preferences node, Display node, etc.

Challenging Dependencies

Several nodes use JSONata, a query language for JSON. This looks like a substantial effort to support and is perhaps impractical on a constrained embedded device. Fortunately, it seems like the Function object can do the same, just less conveniently.

The Template node uses mustache.js for powerful string substitution. Like JSONata, this could be impractical to support on embedded. A small subset is probably straightforward to support, if that would be useful.

The JSON node has an option to use JSON Schema for validation.

Thank You

This exploration was motivated by an extended conversation with Nick O'Leary who patiently explained Node-RED to me at OpenJS World 2022. That clear and patient discussion gave me grounding to begin this effort.

mtoko/node-red-mcu

Node-RED MCU Edition

Introduction

Method

HTTP Request

Inject

Change

Results

Ways to Help

Running Flows on an MCU

Structure

Nodes

Comment

Debug

Function

Inject

Link Call

Link In

Link Out

Catch

Status

GPIO In

GPIO Out

DS18B20

MQTT Broker

MQTT In

MQTT Out

HTTP Request

WebSocket Client

WebSocket In

WebSocket Out

Range

Change

Switch

Filter

Split

JSON

File Write

File Read

MCU Sensor

Delay

Compatibility Node

Future Work

Transformation

Runtime

Nodes

Challenging Dependencies

Thank You