Some measurement for the new broker concept in CAF. These measurements are based on the branch new broker experiments. Since the branch will likely be merged into master at some point, there is a archive of the
- CAF (caf.tar.gz)
- Google Benchmark
- Cmake
- C++ 11 compiler
- Mininet
We used the VM available on the mininet website to run the benchmarks. CAF and google benchmark can be be build with the script setup.sh
in the repository. The VM was configured to use 4 processors and 4096 MB base memory.
The plots were created with R and exported as tikz for the paper (found in the evaluation
folder). When using RStudio, set the working directory to the root folder of this repository and the scripts should find the files to read. (They will also produce PDFs, but the scripts were adjusted to the tikz output and some of the legends might look a little bit misplaced.)
This benchmark uses google benchmark for the measurements. It evaluates the costs of a few layers we implemented for UDP and/or TCP. Here's the command to take the measurements:
$ ./build/bin/layers --benchmark_repetitions=10 --benchmark_report_aggregates_only=true --benchmark_out_format=csv --benchmark_out=evaluation/layers.csv
The resulting csv file can be plotted with the script layers.R
found in the evaluation folder.
This benchmark lets two brokers exchange messages over a link with increasing loss. There is a python script to run the mininet benchmarks, evaluation/mininet.py
. It has several options to determine which benchmarks to run:
$ ./evaluation/mininet.py -h
usage: mininet.py [-h] [-l LOSS] [-d DELAY] [-r RUNS] [-T THREADS] [-R RTO]
[-o] (-t | -u | -q)
CAF newbs on Mininet.
optional arguments:
-h, --help show this help message and exit
-l LOSS, --loss LOSS set packet loss in percent (0)
-d DELAY, --delay DELAY
set link delay in ms (0)
-r RUNS, --runs RUNS set number of runs (1)
-T THREADS, --threads THREADS
set number of threads (1)
-R RTO, --rto RTO set min rto for TCP (40)
-o, --ordered enable ordering for UDP
-t, --tcp use TCP
-u, --udp use UDP
-q, --quic use QUIC
A batch of results could be created as follows:
$ # without delay
$ for i in {0..10}; sudo ./mininet.py -t -l $i -r 10 ; done # TCP
$ for i in {0..10}; sudo ./mininet.py -u -l $i -r 10 ; done # reliable UDP
$ for i in {0..10}; sudo ./mininet.py -u -l $i -r 10 -o ; done # reliable UDP + ordering
$ # with 10ms delay
$ for i in {0..10}; sudo ./mininet.py -t -l $i -r 10 -d 10
$ for i in {0..10}; sudo ./mininet.py -u -l $i -r 10 -d 10
$ for i in {0..10}; sudo ./mininet.py -u -l $i -r 10 -o -d 10
The script stores benchmark output in ./pingpong/
using the nameing scheme $PROTOCOL-{client,server}-$LOSS-$DELAY-$RUN.{out,err}
. Results will be stored in the .out
file of the client. Aggregateing the results into .csv
files can be done with the script evaluation/pingpong/merge_logs.sh
(there is a variable in it to determine which delay measurements to aggregate). It outputs a number of csv files that should contain one colums for the loss percentage and 10 columns for measurements from 0% loss to 10% loss. It expects the requires files, i.e., the .out
logs for loss 0% to 10%, in its folder.
Plots in tikz format can be created with the script pingpong-{0,10}.R
using the previously created csv data.