Benchmarking scripts and results for abusim and goabu.
The results of this benchmark can be found in boxplot form in the ./graphs
directory, and in csv form in the ./csv
directory.
Metrics with description:
received bytes
: Total received bytes per agentreceived packets
: Total received packets per agentsent bytes
: Total sent bytes per agentsent packets
: Total sent packets per agentreceived bytes per seconds
: Like above, but divided by running timereceived packets per seconds
: Like above, but divided by running timesent bytes per seconds
: Like above, but divided by running timesent packets per seconds
: Like above, but divided by running timebytes ratio
: received/sent bytespackets ratio
: received/sent packetsbytes per packet
: Average size of each packettotal bytes
: Total bytes sent in the network by all agentstotal packets
: Total packets sent in the network by all agentstotal bytes per seconds
: Like above, but divided by running timetotal packets per seconds
: Like above, but divided by running timelogs_expected bytes
: Sent bytes as reported in the logslogs_actual bytes
: Sent bytes as measured by the systemlogs_expected packets
: Sent packets as reported in the logslogs_actual packets
: Sent packets as measured by the systemlogs_bytes ratio
: actual/expected byteslogs_packets ratio
: actual/expected packets
Prerequisites:
- Python
- Awk (or some degree of patience)
- Docker
- Any linux system (the script accesses some linux pseudofiles)
- The following docker containers
abulang/abusim-gui:latest
abulang/abusim-coordinator:latest
- This container (on branch
log-sends
), possibly tagged asabulang/abusim-goabu-agent-log:latest
- You can use any agent image that uses this commit of goabu or a later one
- Optionally aeg to generate test cases (requires a rust toolchain for compiling)
Read every section in order before running a benchmark.
The original test cases can be found in the ./test_configs
directory, but new can be created using aeg
, full instructions are found on it's GitHub page.
Remember to set the agent image as abulang/abusim-goabu-agent-log:latest
(or whatever tag you set) or you won't be able to inspect log metrics.
The original were created with:
aeg -a 10 -o idle10.yml --devices-length=3 --image="abulang/abusim-goabu-agent-log:latest"
aeg -a 50 -o idle50.yml --devices-length=3 --image="abulang/abusim-goabu-agent-log:latest"
aeg -a 100 -o idle100.yml --devices-length=3 --image="abulang/abusim-goabu-agent-log:latest"
aeg -a 10 -b 10 -c 5 -d 5 --devices-length=20 -o low10.yml --image="abulang/abusim-goabu-agent-log:latest"
aeg -a 50 -b 10 -c 5 -d 5 --devices-length=20 -o low50.yml --image="abulang/abusim-goabu-agent-log:latest"
aeg -a 100 -b 10 -c 5 -d 5 --devices-length=20 -o low100.yml --image="abulang/abusim-goabu-agent-log:latest"
aeg -a 10 -e=3 --devices-length=5 -o high10.yml --image="abulang/abusim-goabu-agent-log:latest"
aeg -a 50 -e=3 --devices-length=5 -o high50.yml --image="abulang/abusim-goabu-agent-log:latest"
aeg -a 100 -e=3 --devices-length=5 -o high100.yml --image="abulang/abusim-goabu-agent-log:latest"
To launch abusim without logging use:
./abusim -c <path to config file> -g up
To launch abusim with logging use the following command, and from the web GUI (at https://localhost:8080) manually set the log level to "debug" for one of the agents, the test were run logging 'agent0'.
./abusim -c <path to config file> -g up | tee <path to log file>
In case of failed start you might need to manually tear down the environment before retrying, using:
./abusim -c <path to config file> -g down
To run tests with more than ~35 agents you might need to edit your /etc/sysctl.conf
file and increase the values of the net.ipv4.neigh.default.gc_thresh{1,2,3}
variables and reload/reboot. The tests were run win the following values (maximum):
net.ipv4.neigh.default.gc_thresh1 = 2147483647
net.ipv4.neigh.default.gc_thresh2 = 2147483647
net.ipv4.neigh.default.gc_thresh3 = 2147483647
net.ipv6.neigh.default.gc_thresh1 = 2147483647
net.ipv6.neigh.default.gc_thresh2 = 2147483647
net.ipv6.neigh.default.gc_thresh3 = 2147483647
Logs don't provide useful information for idle benchmarking, don't bother collect those logs.
To run the idle tests, launch abusim as explained above and run:
./idle_inspect.py <namespace> <number of devices> <running time> <iterations> <output file>
If you didn't specify otherwise, the namespace will be 'abusim-example'. The tests were run with 500 seconds of running time and 5 iterations.
All tests should have the same number of iterations, later scripts make this assumption.
A running benchmark requires a bit more work.
Before the test you need to discover how long it takes for one computation to end, because the overhead of the simulator causes the actual running times to be longer than expected when increasing the number of devices. For that you will need to launch normally abusim, then start a stopwatch and send to any agent the command start_all = true
(from the GUI) and wait for every queue to empty.
Repeat this for every configuration and write down every running time.
Once you know every running time, relaunch again the configuration you want to benchmark (eventually with logging) and once it's up run:
./running_inspect.py <namespace> <number of devices> <measured running time> <number of iterations> <output file>
The output file of the original benchmarks (found in the ./results
directory, alongside log files) are in the form type_<devices>_<time>_<iterations>
. The configuration were chosen so that the running time with a few devices would be around 500 seconds on a certain machine and then the number was increased to 10, 50 and 100. The benchmarks with more devices required much more time.
All tests (even idle ones) should have the same number of iterations, later scripts make this assumption.
The data digestion script requires the logs of each benchmark to be split in multiple files, one for each iteration. The files must to be in separate directories (the split logs from original test, these directories can be found in the ./results/log-digest
directory), and the files from each iteration must be named 1.log
to n.log
(not from 0).
To split the files you can just use the awk command:
awk 'BEGIN {count=0; filename="0.log"} /Input: start_all = true/{filename=++count".log"}; {print >filename}' <path to log file to split>
This awk command will generate all the files (splitting at the "start_all = true" command which starts the computation) and a file named 0.log
which doesn't contain useful information and will be ignored by the next script, you can delete it.
The last step is reading all the data produced and turing it into useful CSVs and graphs with the ./graph.py
script.
All the filenames and all the information needed for the digestion are hardcoded into the first lines of the script file (from 9 to 30). Before running the script remember to edit those lines it according to your benchmark setup.
Make sure to "align" the data in the lists, e.g.: the first element of the all_files
list must refer to the same benchmark as the first element of the all_times
list.
Each parameter is commented with its description.
Once everything is setup you can digest the data by running:
./graph.py
The results will appear in the ./csv
and ./graphs
directories. It might return an error if such directories don't exist.
This script can take a couple seconds to run.