joosthooz/arrow-benchmarks-ci

Benchmarks CI for Apache Arrow project

arrow-benchmarks-ci

Arrow Benchmarks CI is responsible for

• orchestrating and prioritizing benchmark builds for Apache Arrow based on the list of benchmarks in Benchmarks repo
• publishing benchmark results to Apache Arrow pull requests associated with commits to master branch (example)
• posting benchmarks results to Conbench

Arrow Benchmarks CI consists of

• Buildkite pipelines and scripts for running benchmarks on benchmark machines
• Arrow BCI API service responsible for
  • listening to apache/arrow Pull Request comment events so benchmark builds can be scheduled for pull requests with an @ursabot please benchmark comment
  • persisting benchmark build statistics (e.g., memory usage, run time, errors, and conda packages)

Arrow Benchmarks CI Public Buildkite Pipelines

• Arrow BCI Test is used for testing Arrow BCI API service and CI scripts
• Arrow BCI Benchmark on ec2-t3-xlarge-us-east-2 is used for running cloud benchmarks
• Arrow BCI Benchmark on ursa-i9-9960x is used for running Python, R and JavaScript benchmarks
• Arrow BCI Benchmark on ursa-thinkcentre-m75q is used for running C++ and Java benchmarks
• Arrow BCI Benchmark on voltron-pavilion is not used yet but will be running R TPCH benchmarks in a very near future

Note that you can view builds for these pipelines but you can not manually schedule new builds.

How can I add my own benchmark machine to Arrow Benchmarks CI

Benchmark machines should be bare metal machines dedicated to only running benchmarks to avoid high variability in benchmark results that can result in false regression/improvements.

How to Add New Benchmark Machine

How can I test benchmark builds that run on ursa-i9-9960x and ursa-thinkcentre-m75q locally

# Set env vars for conbench credentials if you need benchmark results to be posted to Conbench during testing
export CONBENCH_EMAIL=<conbench_user_id>
export CONBENCH_PASSWORD=<conbench_user_password>

# Build docker image with Arrow dependencies installed
cd ~/arrow-benchmarks-ci
docker build -f buildkite/benchmark-test/Dockerfile . -t benchmark-test

# Run benchmarks
docker run -i \
    --env BENCHMARKABLE=${BENCHMARKABLE:-"641554b0bcce587549bfcfd0cde3cb4bc23054aa"} \
    --env BENCHMARKABLE_TYPE=${BENCHMARKABLE_TYPE:-"arrow-commit"} \
    --env BENCHMARKS_DATA_DIR="/data" \
    --env CONBENCH_EMAIL=$CONBENCH_EMAIL \
    --env CONBENCH_PASSWORD="$CONBENCH_PASSWORD" \
    --env CONBENCH_URL="https://conbench.ursa.dev" \
    --env MACHINE="docker-container-for-testing-benchmark-builds" \
    --env PYTHON_VERSION=${PYTHON_VERSION:-"3.8"} \
    --env RUN_ID=$BUILDKITE_BUILD_ID \
    --env RUN_NAME=${RUN_NAME:-"benchmark build test: $BUILDKITE_BUILD_ID"} \
    benchmark-test bash buildkite/benchmark/utils.sh build_arrow_and_run_benchmark_groups

How can I test benchmark builds that run on ursa-i9-9960x and ursa-thinkcentre-m75q using Buildkite

This option is only available to Apache Arrow CI Buildkite org users at the moment.

1. Go to https://buildkite.com/apache-arrow/arrow-bci-benchmark-build-test
2. Click New Build
   • Set Commit and Branch for arrow-benchmarks-ci repo that you would like to test
   • Click Create Build