benchdiff

Benchdiff runs go benchmarks on your current git worktree and a base ref then uses benchstat to show the delta.

What it does

Benchdiff automates comparing the results of the same go benchmarks on your worktree and a git reference.

It essentially combines multiple git, go test -bench and benchstat commands into one.

These are the basic steps benchdiff performs:

Runs go test -bench ... on your current worktree and saves the results to a cache directory.
Creates a new worktree at the base ref
Runs go test -bench ... on the base worktree and saves the results to cache again.
Runs benchstat to compare the base and head results from cache.

GitHub Action

benchdiff can be run as a GitHub Action. https://github.com/WillAbides/benchdiff-action

Usage

`benchdiff --help` output

Usage: benchdiff

benchdiff runs go benchmarks on your current git worktree and a base ref then uses benchstat to show
the delta.

More documentation at https://github.com/willabides/benchdiff.

Flags:
  -h, --help       Show context-sensitive help.
      --version    Output the benchdiff version and exit.
      --debug      write verbose output to stderr

  --base-ref="HEAD"    The git ref to be used as a baseline.
  --cooldown=100ms     How long to pause for cooldown between head and base runs.
  --force-base         Rerun benchmarks on the base reference even if the output already exists.
  --git-cmd="git"      The executable to use for git commands.
  --json               Format output as JSON.
  --on-degrade=0       Exit code when there is a statistically significant degradation in the
                       results.
  --tolerance=10.0     The minimum percent change before a result is considered degraded.

benchmark command line
  --bench="."              Run only those benchmarks matching a regular expression. To run all
                           benchmarks, use '--bench .'.
  --benchmark-args=args    Override the default args to the go command. This may be a template.
                           See https://github.com/willabides/benchdiff for details."
  --benchmark-cmd="go"     The command to use for benchmarks.
  --benchmem               Memory allocation statistics for benchmarks.
  --benchtime=STRING       Run enough iterations of each benchmark to take t, specified as a
                           time.Duration (for example, --benchtime 1h30s). The default is 1 second
                           (1s). The special syntax Nx means to run the benchmark N times (for
                           example, -benchtime 100x).
  --count=10               Run each benchmark n times. If --cpu is set, run n times for each
                           GOMAXPROCS value.'
  --cpu=GOMAXPROCS,...     Specify a list of GOMAXPROCS values for which the benchmarks should be
                           executed. The default is the current value of GOMAXPROCS.
  --packages="./..."       Run benchmarks in these packages.
  --show-bench-cmdline     Instead of running benchmarks, output the command that would be used and
                           exit.
  --tags=STRING            Set the -tags flag on the go test command
  --warmup-count=INT       Run benchmarks with -count=n as a warmup
  --warmup-time=STRING     When warmups are run, set -benchtime=n

benchstat options
  --alpha=0.05                 consider change significant if p < α
  --benchstat-output="text"    format for benchstat output (csv,html,markdown or text)
  --delta-test="utest"         significance test to apply to delta: utest, ttest, or none
  --geomean                    print the geometric mean of each file
  --norange                    suppress range columns (CSV and markdown only)
  --reverse-sort               reverse sort order
  --sort="none"                sort by order: delta, name, none
  --split="pkg,goos,goarch"    split benchmarks by labels

benchmark result cache
  --cache-dir=STRING    Override the default directory where benchmark output is kept.
  --clear-cache         Remove benchdiff files from the cache dir.
  --show-cache-dir      Output the cache dir and exit.

`--benchmark-args`

The default should work for most cases.

--benchmark-args is everything after "go" in the go test command that runs the benchmarks. You can use either pass in the raw values or use a go template.

Default value:

test {{ .Packages }} -run '^$'
{{- if .Bench }} -bench {{ .Bench }}{{end}}
{{- if .Count }} -count {{ .Count }}{{end}}
{{- if .Benchtime }} -benchtime {{ .Benchtime }}{{end}}
{{- if .CPU }} -cpu {{ .CPU }}{{ end }}
{{- if .Tags }} -tags "{{ .Tags }}"{{ end }}
{{- if .Benchmem }} -benchmem{{ end }}

Install

go get

go get -u github.com/willabides/benchdiff/cmd/benchdiff

bindown

Add a bindown dependency:

$ bindown template-source add benchdiff https://raw.githubusercontent.com/WillAbides/benchdiff/main/bindown.yml
$ bindown dependency add benchdiff benchdiff#benchdiff
Please enter a value for required variable "version":	<latest version>

FiloSottile/benchdiff