temci

An advanced benchmarking tool written in Python 3 that supports setting up an environment for benchmarking and the generation of visually appealing reports.

It runs on Linux systems and (rudimentarily) on macOS.

Why should you use temci?

temci allows you to easily measure the execution time (and other things) of programs and compare them against each other resulting in a pretty HTML5 based report. Furthermore it can set up the environment to ensure benchmarking results with a low variance. The latter feature can be used without using temci for benchmarking by using temci short shell.

Usage

The main commands of temci are temci exec and temci report.

Suppose you want to see whether grepping for the strings that consist of a and b in the current folder is slower than for strings that consist only of a.

First we have to install temci (using Nix, see below for more instructions):

nix-env -f https://github.com/parttimenerd/temci/archive/master.tar.gz -i

After this, we can benchmark both commands with temci:

# benchmark both commands 20 times
temci short exec "grep '[ab]*' -R ." "grep 'a*' -R ." --runs 10

# append --watch to get report (in which you can move with the arrow keys and scroll)
# after every benchmark completed (use --watch_every to decrease interval)
temci short exec "grep '[ab]*' -R ." "grep 'a*' -R ." --runs 10 --watch

# if you want to improve the stability your benchmarks, run them with root privileges
# the benchmarked programs are run with your current privileges
temci short exec "grep '[ab]*' -R ." "grep 'a*' -R ." --runs 10 --sudo --preset usable

This results in a run_output.yaml file that should look like:

- attributes: {description: 'grep ''[ab]*'' -R .'}
  data:
    etime: [0.03, 0.02, 0.02, 0.03, 0.03, 0.03, 0.02, 0.03, 0.03, 0.02]
    … # other properties
- attributes: {description: grep 'a*' -R .}
  data:
    etime: [0.02, 0.03, 0.02, 0.03, 0.03, 0.02, 0.03, 0.03, 0.02, 0.02]
    … # other properties
- property_descriptions: {etime: elapsed real (wall clock) time, … }

For more information on the support measurement tools (like perf stat and rusage), the supported plugins for setting up the environment and more, see temci exec.

We can now create a report from these benchmarking results using temci report. We use the option --properties to include only the elapsed time in the report to keep the report simple:

> temci report run_output.yaml --properties etime
Report for single runs
grep '[ab]*' -R .    (   10 single benchmarks)
     etime mean =     2(6).(000)m, deviation = 18.84223%

grep 'a*' -R .       (   10 single benchmarks)
     etime mean =     2(5).(000)m, deviation = 20.00000%

Equal program blocks
     grep '[ab]*' -R .  ⟷  grep 'a*' -R .
         etime confidence =        67%, speed up =      3.85%

We see that there is no significant difference between the two commands.

There are multiple reporters besides the default console reporter. Another reporter is the html2 reporter that produces an HTML report, use it by adding the --reporter html2 option:

http://mostlynerdless.de/files/report_readme/html_report.png

Installation

The simplest way is to use the Nix package manager, after installing Nix, run:

nix-env -f https://github.com/parttimenerd/temci/archive/master.tar.gz -i

Using pip requiring at least Python 3.6:

sudo pip3 install temci

For more information see the Installation page.

Auto completion

Temci can generate auto completion files for bash and zsh. Add the following line to your .bashrc or .zshrc:

. `temci_completion $0`

Using temci to set up a benchmarking environment

Use the temci short shell COMMAND to run a command (sh by default) in a shell that is inside the benchmarking environment. Most options of temci short exec are supported. For more information, see temci shell.

Why is temci called temci?

The problem in naming programs is that most good program names are already taken. A good program or project name has (in my opinion) the following properties:

it shouldn't be used on the relevant platforms (in this case: github and pypi)
it should be short (no one wants to type long program names)
it should be pronounceable
it should have at least something to do with the program

temci is such a name. It's lojban for time (i.e. the time duration between two moments or events).

Contributing

Bug reports and code contributions are highly appreciated.

For more information, see the Contributing page.