BenchmarkTools.jl

BenchmarkTools makes performance tracking of Julia code easy by supplying a framework for writing and running groups of benchmarks as well as comparing benchmark results.

This package is used to write and run the benchmarks found in BaseBenchmarks.jl.

The CI infrastructure for automated performance testing of the Julia language is not in this package, but can be found in Nanosoldier.jl.

Installation

To install BenchmarkTools, you can run the following:

Pkg.add("BenchmarkTools")

Documentation

If you're just getting started, check out the manual for a thorough explanation of BenchmarkTools.

If you want to explore the BenchmarkTools API, see the reference document.

If you want a short example of a toy benchmark suite, see the sample file in this repo (benchmark/benchmarks.jl).

If you want an extensive example of a benchmark suite being used in the real world, you can look at the source code of BaseBenchmarks.jl.

If you're benchmarking on Linux, I wrote up a series of tips and tricks to help eliminate noise during performance tests.

Quick Start

The simplest usage is via the @btime macro, which is analogous to Julia's built-in @time macro but is often more accurate (by collecting results over multiple runs):

julia> using BenchmarkTools

julia> @btime sin(1)
  15.081 ns (0 allocations: 0 bytes)
0.8414709848078965

If the expression you want to benchmark depends on external variables, you should use $ to "interpolate" them into the benchmark expression to avoid the problems of benchmarking with globals. Essentially, any interpolated variable $x or expression $(...) is "pre-computed" before benchmarking begins:

julia> A = rand(3,3);

julia> @btime inv($A);            # we interpolate the global variable A with $A
  1.191 μs (10 allocations: 2.31 KiB)
  
julia> @btime inv($(rand(3,3)));  # interpolation: the rand(3,3) call occurs before benchmarking
  1.192 μs (10 allocations: 2.31 KiB)
  
julia> @btime inv(rand(3,3));     # the rand(3,3) call is included in the benchmark time
  1.295 μs (11 allocations: 2.47 KiB)

As described the manual, the BenchmarkTools package supports many other features, both for additional output and for more fine-grained control over the benchmarking process.

Why does this package exist?

Our story begins with two packages, "Benchmarks" and "BenchmarkTrackers". The Benchmarks package implemented an execution strategy for collecting and summarizing individual benchmark results, while BenchmarkTrackers implemented a framework for organizing, running, and determining regressions of groups of benchmarks. Under the hood, BenchmarkTrackers relied on Benchmarks for actual benchmark execution.

For a while, the Benchmarks + BenchmarkTrackers system was used for automated performance testing of Julia's Base library. It soon became apparent that the system suffered from a variety of issues:

Individual sample noise could significantly change the execution strategy used to collect further samples.
The estimates used to characterize benchmark results and to detect regressions were statistically vulnerable to noise (i.e. not robust).
Different benchmarks have different noise tolerances, but there was no way to tune this parameter on a per-benchmark basis.
Running benchmarks took a long time - an order of magnitude longer than theoretically necessary for many functions.
Using the system in the REPL (for example, to reproduce regressions locally) was often cumbersome.

The BenchmarkTools package is a response to these issues, designed by examining user reports and the benchmark data generated by the old system. BenchmarkTools offers the following solutions to the corresponding issues above:

Benchmark execution parameters are configured separately from the execution of the benchmark itself. This means that subsequent experiments are performed more consistently, avoiding branching "substrategies" based on small numbers of samples.
A variety of simple estimators are supported, and the user can pick which one to use for regression detection.
Noise tolerance has been made a per-benchmark configuration parameter.
Benchmark configuration parameters can be easily cached and reloaded, significantly reducing benchmark execution time.
The API is simpler, more transparent, and overall easier to use.

Acknowledgements

This package was authored primarily by Jarrett Revels (@jrevels). Additionally, I'd like to thank the following people: