For documentation about how this works, see the developer docs.
Visit the benchmark action and click the "Run Workflow" button.
The available parameters are:
fork: The fork of CPython to benchmark. If benchmarking a pull request, this would normally be your GitHub username.ref: The branch, tag or commit SHA to benchmark. If a SHA, it must be the full SHA, since finding it by a prefix is not supported.machine: The machine to run on. One oflinux-amd64(default),windows-amd64,darwin-arm64orall.benchmark_base: If checked, the base of the selected branch will also be benchmarked. The base is determined by runninggit merge-base upstream/main $ref.publish: If checked, the results will be published in the public ideas repo upon successful completion.
To watch the progress of the benchmark, select it from the benchmark action page. It may be canceled from there as well. To show only your benchmark workflows, select your GitHub ID from the "Actor" dropdown.
When the benchmarking is complete, the results are published to this repository and will appear in the master table. Each set of benchmarks will have:
- The raw
.jsonresults from pyperformance. - Comparisons against important reference releases, as well as the merge base of the branch if
benchmark_basewas selected. These include- A markdown table produced by
pyperf compare_to. - A set of "violin" plots showing the distribution of results for each benchmark.
- A markdown table produced by
The most convenient way to get results locally is to clone this repo and git pull from it.
To automate benchmarking runs, it may be more convenient to use the GitHub CLI.
Once you have gh installed and configured, you can run benchmarks by cloning this repository and then from inside it:
$ gh workflow run benchmark.yml -f fork=me -f ref=my_branchAny of the parameters described above are available at the commandline using the -f key=value syntax.
The following is only a summary of certain key revisions. There is also a complete list of results.
| date | fork | ref | version | hash | vs. 3.10.4: | vs. 3.11.0: | vs. base: |
|---|---|---|---|---|---|---|---|
| 2022-03-23 | python | main | 3.10.4 | 9d38120 | 1.26x slower * | ||
| 2022-12-06 | python | 1dd9be6584 | 3.10.9 | 1dd9be6 | 1.01x slower * | 1.26x slower | |
| 2021-10-05 | python | 7c12e4835e | 3.11.0a1 | 7c12e48 | 1.12x faster * | 1.12x slower | |
| 2021-11-05 | python | e2b4e4bab9 | 3.11.0a2 | e2b4e4b | 1.15x faster * | 1.09x slower | |
| 2021-12-08 | python | main | 3.11.0a3 | 2e91dba | 1.19x faster | 1.06x slower * | |
| 2022-01-13 | python | main | 3.11.0a4 | 9471106 | 1.21x faster | 1.04x slower * | |
| 2022-02-03 | python | main | 3.11.0a5 | c4e4b91 | 1.21x faster | 1.04x slower * | |
| 2022-03-07 | python | main | 3.11.0a6 | 3ddfa55 | 1.19x faster | 1.06x slower * | |
| 2022-04-05 | python | main | 3.11.0a7 | 2e49bd0 | 1.23x faster | 1.02x slower * | |
| 2022-05-06 | python | main | 3.11.0b1 | 8d32a5c | 1.28x faster | 1.01x faster * | |
| 2022-05-30 | python | main | 3.11.0b2 | 72f00f4 | 1.28x faster | 1.02x faster * | |
| 2022-06-01 | python | main | 3.11.0b3 | eb0004c | 1.29x faster | 1.02x faster * | |
| 2022-07-11 | python | 5a7e1e0a92 | 3.11.0b4 | 5a7e1e0 | 1.27x faster * | 1.01x faster | |
| 2022-07-25 | python | 0771d71eea | 3.11.0b5 | 0771d71 | 1.27x faster * | 1.01x faster | |
| 2022-08-05 | python | 41cb07120b | 3.11.0rc1 | 41cb071 | 1.27x faster * | 1.00x faster | |
| 2022-09-11 | python | ed7c3ff156 | 3.11.0rc2 | ed7c3ff | 1.26x faster * | 1.00x slower | |
| 2022-10-24 | python | v3.11.0 | 3.11.0 | deaf509 | 1.26x faster * | ||
| 2022-12-06 | python | a7a450f84a | 3.11.1 | a7a450f | 1.25x faster * | 1.00x slower | |
| 2022-10-22 | python | main | 3.12.0a1+ | f58631b | 1.30x faster * | 1.03x faster * | |
| 2022-12-05 | python | e3a3863cb9 | 3.12.0a2+ | e3a3863 | 1.31x faster * | 1.03x faster | |
| 2022-12-21 | python | a7715ccfba | 3.12.0a3+ | a7715cc | 1.32x faster * | 1.03x faster |
| date | fork | ref | version | hash | vs. 3.10.4: | vs. 3.11.0: | vs. base: |
|---|---|---|---|---|---|---|---|
| 2022-03-23 | python | v3.10.4 | 3.10.4 | 9d38120 | 1.22x slower | ||
| 2022-12-06 | python | 1dd9be6584 | 3.10.9 | 1dd9be6 | 1.00x slower | 1.22x slower | |
| 2021-11-05 | python | e2b4e4bab9 | 3.11.0a2 | e2b4e4b | 1.15x faster | 1.05x slower | |
| 2021-12-08 | python | 2e91dba437 | 3.11.0a3 | 2e91dba | 1.15x faster | 1.06x slower | |
| 2022-01-13 | python | 9471106fd5 | 3.11.0a4 | 9471106 | 1.17x faster | 1.04x slower | |
| 2022-02-03 | python | c4e4b91557 | 3.11.0a5 | c4e4b91 | 1.09x faster | 1.12x slower | |
| 2022-03-07 | python | 3ddfa55df4 | 3.11.0a6 | 3ddfa55 | 1.17x faster | 1.04x slower | |
| 2022-04-05 | python | 2e49bd06c5 | 3.11.0a7 | 2e49bd0 | 1.22x faster | 1.01x slower | |
| 2022-05-06 | python | 8d32a5c8c4 | 3.11.0b1 | 8d32a5c | 1.21x faster | 1.01x slower | |
| 2022-05-30 | python | 72f00f420a | 3.11.0b2 | 72f00f4 | 1.21x faster | 1.01x slower | |
| 2022-06-01 | python | eb0004c271 | 3.11.0b3 | eb0004c | 1.21x faster | 1.01x slower | |
| 2022-07-11 | python | 5a7e1e0a92 | 3.11.0b4 | 5a7e1e0 | 1.23x faster | 1.01x faster | |
| 2022-07-25 | python | 0771d71eea | 3.11.0b5 | 0771d71 | 1.23x faster | 1.01x faster | |
| 2022-08-05 | python | 41cb07120b | 3.11.0rc1 | 41cb071 | 1.23x faster | 1.01x faster | |
| 2022-09-11 | python | ed7c3ff156 | 3.11.0rc2 | ed7c3ff | 1.22x faster | 1.00x faster | |
| 2022-10-24 | python | deaf509e8f | 3.11.0 | deaf509 | 1.22x faster | ||
| 2022-12-06 | python | a7a450f84a | 3.11.1 | a7a450f | 1.22x faster | 1.00x faster | |
| 2022-10-24 | python | dfb5d272e6 | 3.12.0a0 | dfb5d27 | 1.20x faster | 1.02x slower | |
| 2022-11-13 | python | 367f552129 | 3.12.0a1+ | 367f552 | 1.19x faster | 1.03x slower | |
| 2022-12-05 | python | e3a3863cb9 | 3.12.0a2+ | e3a3863 | 1.18x faster | 1.04x slower | |
| 2022-12-16 | python | main | 3.12.0a3+ | 432117c | 1.24x faster | 1.02x faster |
* indicates that the exact same versions of pyperformance was not used.