MONDO-SAM reporting

Reporting tools for MONDO-SAM

Prerequisites

Install R and the required packages.

Linux

Install R 3.2+. On Ubuntu 15.10+, this is as simple as issuing:

sudo apt-get install -y r-base r-base-dev
sudo R -f install.R

On older Ubuntu systems, follow the Ubuntu Packages for R guide. For other Linux systems, refer to the CRAN Linux pages.

Windows

Install R and adjust the system environment variables.

In a command prompt with administrator rights, run:

R -f install.R

To edit the script, it is recommended to use RStudio.

Usage

To draw a few simple plots, issue the following command:

./report.sh

The results should be the results.csv file. For a sample CSV file, open the sample-results.csv file with your favorite spreadsheet software or text editor.

The schema attributes are defined in the following list. For each attribute, we list an example from the Train Benchmark framework 🚂.

Scenario
- defines the workflow of the benchmark
- 🚂 the Batch scenario defines a workflow with a single read and query operation.
Tool
- defines the system under benchmark with it's custom settings
- 🚂 EMF-IncQuery (Local Search)
Run
- specifies the index of the run
- 🚂 there are usually 5 runs, so the Run attribute takes a value between 1 and 5
Case
- defines the benchmark case
- 🚂 the query PosLength
Artifact
- defines the artifact which used by the benchmark
- 🚂 the model size used by the queries and transformations.
Phase
- defines the phase of the benchmark
- 🚂 the Read phase loads the model
Metric
- the metric stored by the particular row
- 🚂 the benchmark uses two metrics; Time (for measuring the execution time in nanoseconds) and Matches
Iteration
- defines the index of the current iteration if a step is executed multiple times
- 🚂 for the Transformation and Recheck phases in the Repair scenario, the Iteration attribute takes a value between 1 and 10
Value
- The value of the metric.
- 🚂 for Time, a valid value is 456123123 (for an approximately half-second operation), for Matches, a valid value is 5