This repo contains pipeline processing elements (PPEs) that can be used to create streaming data flow processing pipelines. rtndf uses Manifold (https://github.com/richardstechnotes/Manifold) as the networking infrastructure to form high performance pipelines. PPEs can include machine learning elements based on TensorFlow or any other ML engine.
PPEs can be written in Python and C++ currently.
MIT license.
Using docker builds greatly simplifies host machine dependency issues. There are two directories, DockerBuild and DockerRun, that can be used to build and run dockerized nodes.
DockerBuild contains scripts to build each individual component. The all script will build everything. Sometimes there are multiple version of the different components to add things like Tensorflow support and GPU support.
If the Manifold containers have not been pulled or built locally, copies will be pulled from Docker Hub. In addition, if the rtndfcore containers have not been pulled or built locally, they will also be pulled from Docker Hub.
DockerRun contains scripts that provide an easy way to correctly run dockerized nodes. Extra runtime arguments can be added as runtime arguments to the run script.
ManifoldCore must be built and installed - https://github.com/richardstechnotes/Manifold.
Then rtndf core support must be installed:
cd ~/rtndf/rtndfcore
./build
Each Python node can then be run a bit like this:
cd ~/rtndf/Python/uvccam
python uvccam.py
For C++ nodes, see the readme in the ~/rntdf/Cpp directory.
- uvccam. UVC-based USB webcams can be used to generate MJPEG video streams using this element.
- picam. A Raspberry PI fitted with a PiCamera can be used to generate an MJPEG video stream using this script.
- audio. This element streams PCM audio.
- imu. Streams data from an IMU (e.g. Raspberry Pi fitted with SenseHat).
- sensehat. Streams all sensor data from a Raspberry Pi with SenseHat.
- sensors. Streams sensor data from selected environmental sensors.
- imageproc. A video pipeline element that uses TensorFlow to perform simple image processing - processing a half-sized image with a 3 x 3 Laplacian.
- modet. A video pipeline element that implements a motion detector that draws a box around objects in motion.
- recognize. A video pipeline element that tries to recognize objects in the stream using Inception-v3.
- facerec. A video pipeline element that tries to recognize faces in the stream using OpenFace (https://cmusatyalab.github.io/openface/).
- speechdecode. An audio pipeline processor that uses CMU Sphinx to decode words and phrases in the stream.
- avview. This script can be used to display video streams and play audio streams.
- imuview. An OpenGL node that can display being streamed by the imu PPE.
- sensorview. Display sensor data generated by sensehat and sensors.
- tts. A text to speech node.
- tty. A serial interface node.
Check out https://richardstechnotes.wordpress.com/ for more information.
Installation and use instructions can be found in Python/README.md and Cpp/README.md.
All pipeline messages are encoded using JSON. The basic message includes standard field from rtndfdefs.py, a metadata entry and the data fields.
See Python/rtndf/metadefs/rtndfdefs.py for more information.
All messages include a standard header that include the following fields:
* TIMESTAMP - the time at which the message was generated (in seconds, float).
* DEVICEID - a string indicating the unique ID of the source of the message.
* TOPIC - a string indicating the topic to which this message belongs.
* TYPE - the data of data in the Data field.
* METADATA - this is a JSON object that contains entries from PPEs that have added information to the message.
The PPEs can insert extra metadata based on their processing into the METADATA field. These are described in the JSON defs file for the PPE in the rtndf/metadefs directory.
The Data field contains the actual data part of the message, if there is one. The format of this field depends on the type of data. The formats for this field are described in the Python/rtndf/datadefs directory.