[cargo-]cbench

Environment control for benchmarks on Linux/systemd, reducing external noise from benchmark results.

cbench can be used to run any programs. cargo-cbench is a wrapper giving a cargo bench-like interface for running cargo benches conveniently.

Installation

cargo install cbench

Usage

For Rust projects with cargo benches, simply replace cargo bench with cargo cbench:

cargo cbench

This will setup the environment and tweak system configurables (see the next section), run all cargo benchmarks in the current project, and finally revert changes to the original state.

It will only allocate a single exclusive CPU for the benchmark. If your program is multi-threaded, you can allocate more by:

cargo cbench --cpus=1-2

Cargo flags and bench program flags can be passed using the same syntax:

cargo cbench --bench=bench1 --features=feat1 -- --exact foo

For other benchmarking frameworks, run cbench following by the command:

cbench hyperfine /some/benchee

By default, the target command will be run inside a systemd unit named 'cbench.service' as the current user. It will be in a clean environment without inheriting from the current shell. If your command relies on some environment variables, you need to pass them explicitly via --setenv=ENV or --setenv=ENV=VALUE.

More control arguments can be seen in cbench --help.

What it does

noaslr: Disable Address Space Layout Randomization (ASLR) via /proc/sys/kernel/randomize_va_space
cpuset: Pin the target process' cgroup on specific CPU(s) for exclusive use via cgroup cpuset.
noht: Disable (set offline) CPU thread siblings of the CPU(s) used if hyper-threading is enabled, via CPU hotplug.
cpufreq: Set power governor of target CPU(s) to 'performance' and disable adaptive turbo/boost, via CPU Performance Scaling.
noirq: Mask used CPU(s) from IRQ affinity.

These control modules can be enabled or disabled individually via --with= or --without=.

What it does NOT

We don't do benchmarks, but we setup environment and tunables for benchmark programs to do benchmarks more reliably. It's expected to be used together with benchmark frameworks/programs like criterion or hyperfine.
Environment control does not make programs run faster, but typically in opposite, because we disable frequency boost by default. Our goal is consistency rather than performance.
We reduce external noise from tainting the benchmark results. But we cannot magically stabilize it from internal biases. Benchee may still be unstable under different memory (heap and stack) layout caused by environment variables or "being lucky" on program initialization, producing a seemingly random systemic bias through multiple runs. You need to carefully write your benchee program to reduce this effect.

See stabilizer for more information.

Privileged operations

All settings mentioned above are privileged and machine global. To minimize the impact and security risks, we leverage systemd-run privileged ExecStartPre=/ExecStopPost= commands, thus only the environment setup and reset will be executed with root privileges. Authentication is done by systemd-run itself by default (via PolKit), or you can use --use-sudo to use sudo instead.

Note that no matter whether --use-sudo is used, the program compilation (via cargo build) and benchmark processes are always running as the current user. Never add sudo to cargo cbench itself! Even if you really want to run the target artifact/command as root because, say you are running perf stat --all-kernel, use the option --root for it.

Environment modifications will be reverted after the program exits via systemd ExecStopPost= command, which will do the clean up even if the target process aborted unexpectedly (eg. by Ctrl-C). If they do not, please report a bug.

Credit

Heavily motivated by LLVM benchmarking tips.
Thank QuarticCat@github for turbo/boost control tips.
Thank PeterCxy@github for IRQ control tips.