shotover/shotover-proxy

windsock: implement `--set-baseline` and `--clear-baseline`

Closed this issue · 0 comments

rukai commented

These two flags will be used to enable and disable the baseline feature.
While the baseline feature is disabled every bench run will display a results table with just one column: the results of the bench that just ran.
While the baseline feature is enabled every bench run will display a comparison table of the benches results and its baselines results.
If a baseline for that bench does not exist it will fall back to a regular results table.

To implement baselines we will

  • Store run data at target/windsock_data/last_run instead of target/windsock_data
  • clear the contents of target/windsock_data/last_run at the beginning of every run.
  • take a snapshot of the target/windsock_data/last_run at target/windsock_data/baseline on --set-baseline and delete it on --clear-baseline
  • when running benches if a baseline exists at target/windsock_data/baseline/bench_name form a comparison table using it instead of a results table.
  • Introduce a new flag --compare-against-baseline which forms a comparison table where every column is compared against its previous run.