RMLStreamer benchmark suite

RMLStreamer benchmark consists of 3 major components.

Data streamer
System under test (SUT)
Monitoring unit

All 3 components are setup in a containerized environment using docker. The architecture for the docker setup is influenced by RSPLab [Riccardo T.]. This repo is available as open-source under the MIT License.

Run the evaluation

The following command executes the scripts which will automatically start the data streamer, the specified SUT for evaluation and the collection of the metrics. The script will also ensure that the components have to ensure that the metrics collection do not start too early.

./start.sh -e [SUT folder name] -- [CLI args for SUT]

CLI args for SUT are the args required by the SUT to run a mapping job. Check the documentation of the relevant SUTs for the required arguments to run them.

Components

Information on running the scripts, the configuration and the workings of each components can be found at their relevant pages/repos.

General structure

The components are structured according to the following the file hierarchy:

root 
|  start.sh
|  stop.sh
|_____datastreamer 
|     |   start.sh 
|     |   stop.sh
|     |   ....
|_____engine
|     |   start.sh 
|     |   stop.sh
|     |   ....
|     |_____rml-streamer 
|           |  start.sh 
|           |  stop.sh
|           |  ....
|_____collector
      |   start.sh 
      |   stop.sh
      |   ....

The start.sh scripts are the entry scripts to start the relevant components/engines. The stop.sh scripts starts the shutdown procedure for the relevant components/engines. It should also be responsible for the removal of any resources used by the components.

Adding your own engine to the benchmark

In order to add another engine to evaluate with this benchmark framework, add another engine-specific folder under the engine folder. The entry script to start the engine should be named start.sh.

Other infos

Wait-for-it.sh

The wait-for-it.sh script will block the execution of script wth a timeout waiting for the specified port at the given host to be open. The script can be run as follows: ./wait-for-it.sh [hostname:port] -t [seconds]