Bencher

Bencher is a suite of continuous benchmarking tools designed to catch performance regressions in CI. That is, Bencher allows you to detect and prevent performance regressions before they make it to production.

For the same reasons that unit tests are run in CI to prevent feature regressions, benchmarks should also be run in CI to prevent performance regressions. Performance bugs are bugs!

Bencher consists of:

bencher CLI
Bencher API Server
Bencher Web UI

The best place to start is the Bencher Quick Start tutorial.

Though Bencher is open source, there is also a hosted version available Bencher Cloud.

🐰 Use the GitHub Action with your project

Documentation

Supported Benchmark Harnesses

{...} JSON
- Custom benchmark harness support
#️⃣ C#
- BenchmarkDotNet
➕ C++
- Catch2
- Google Benchmark
🕳 Go
- go test -bench
☕️ Java
- Java Microbenchmark Harness (JMH)
🕸 JavaScript
- Benchmark.js
- console.time/console.timeEnd
🐍 Python
- airspeed velocity
- pytest-benchmark
♦️ Ruby
- Benchmark
🦀 Rust
- libtest bench
- Criterion

For more details see the explanation of benchmark harness adapters.

Share Your Benchmarks

All public projects have their own perf page. These results can easily be shared with an auto-updating perf image. Perfect for your README!

GitHub Actions

Install the Bencher CLI using the GitHub Action. Use it to track your benchmarks. See how to use GitHub Actions for more details.

name: Track benchmarks with Bencher
on: [push]
jobs:
  benchmark_with_bencher:
    name: Benchmark with Bencher
    runs-on: ubuntu-latest
    env:
      - BENCHER_API_TOKEN: ${{ secrets.BENCHER_API_TOKEN }}
    steps:
      - uses: actions/checkout@v3
      - uses: bencherdev/bencher@v0.2.42
      - run: bencher run --project my-project-slug "bencher mock"

Repository Secrets

Add BENCHER_API_TOKEN to you Repository secrets (ex: https://github.com/my-user-slug/my-repo/settings/secrets/actions). You can find your API tokens by running bencher token ls --user my-user-slug or by going to the Bencher Console (ex: https://bencher.dev/console/users/my-user-slug/tokens).

Specify CLI Version

- uses: bencherdev/bencher@v0.2.42

There is also an optional version argument to specify an exact version of the Bencher CLI to use. Otherwise, it will default to using the latest CLI version.

- uses: bencherdev/bencher@main
  with:
    version: 0.2.42

These two commands are functionally equivalent. However, the first protects you from any breaking changes in the Bencher CLI GitHub action by using a tagged release. Therefore, the former is the suggested method (ex: - uses: bencherdev/bencher@v0.2.42)

Contributing

The easiest way to contribute is to open the repo in GitPod. Everything you need will already be there! It is best to connect to the GitPod instance with VS Code Desktop via SSH. There is a hotkey set to tapping caps lock twice to prompt the web VS Code to create a VS Code Desktop SSH session. For more details see the GitPod docs here. Once set up, both the UI and API should be built, running, and seeded at localhost:3000 and localhost:61016 respectively.

License

All content that resides under any directory or feature named "plus" is licensed under the Bencher Plus License.

All other content is license under either of Apache License, Version 2.0 or MIT license at your discretion.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Bencher by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

osiewicz/bencher