CI Benchmarking reports opposite results
sbryngelson opened this issue · 5 comments
The CI Benchmarker reports master is X times faster than PR (or something like this) but in reality it is actually reporting master is X times slower than PR. This is a lhs vs rhs swap issue in bench.py.
we were interpreting the results incorrectly, and the current message is technically correct but confusing. a more clear wording of this message would be helpful (or just an additional sentence like 1.5x indicates the PR is twice as fast as Master, 0.5x the PR is twice as slow. @anandrdbz
Hmm I guess the problem is either way you're gonna run into an issue when x < 1, since it's much more natural to interpret values > 1. And the problem is this line doesn't actually use a value for x, it is simply written to indicate what the table values are.
We could add a comment/line before the table that reads
For example: 1.5x indicates the PR is 1.5-times faster than Master; 0.5x indicates the PR is 0.5-times as fast as Master (so, slower)
Okay I can do that, and maybe we should do it from the perspective of the PR anyways since master is the baseline (so PR is x times slower/faster than master instead of master is x times slower/faster than PR)
That's fine as long as it's correct and is instructive