windsock: implement `--set-baseline` and `--clear-baseline`
Closed this issue · 0 comments
rukai commented
These two flags will be used to enable and disable the baseline feature.
While the baseline feature is disabled every bench run will display a results table with just one column: the results of the bench that just ran.
While the baseline feature is enabled every bench run will display a comparison table of the benches results and its baselines results.
If a baseline for that bench does not exist it will fall back to a regular results table.
To implement baselines we will
- Store run data at
target/windsock_data/last_run
instead oftarget/windsock_data
- clear the contents of
target/windsock_data/last_run
at the beginning of every run. - take a snapshot of the
target/windsock_data/last_run
attarget/windsock_data/baseline
on --set-baseline and delete it on--clear-baseline
- when running benches if a baseline exists at
target/windsock_data/baseline/bench_name
form a comparison table using it instead of a results table. - Introduce a new flag
--compare-against-baseline
which forms a comparison table where every column is compared against its previous run.