sharkdp/hyperfine

Feature request: Support combining benchmark results

Opened this issue · 0 comments

The following use case happened often enough to trigger me to write this feature request: I often want to benchmark a modification made to the system, but this modification would need some effort to script. So I would like to call hyperfine, then modify the system, then call hyperfine again. Instead of having the result of two individual results, I like to have a nice comparison of those two.

Recent example: I wanted to benchmark the performance impact of the patch submitted in dracut-ng/dracut-ng#1678. So I created a chroot with Dracut installed. Then I ran hyperfine "dracut --debug --force" for the baseline result. Then I applied the submitted patch to /usr/lib/dracut/dracut-logger.sh and ran hyperfine "dracut --debug --force" again. Now I have those results and have to calculate the difference myself:

# hyperfine "dracut --debug --force"
Benchmark 1: dracut --debug --force
  Time (mean ± σ):      8.427 s ±  0.145 s    [User: 12.157 s, System: 4.686 s]
  Range (min … max):    8.208 s …  8.745 s    10 runs
 
# hyperfine "dracut --debug --force"
Benchmark 1: dracut --debug --force
  Time (mean ± σ):      8.306 s ±  0.071 s    [User: 11.887 s, System: 4.624 s]
  Range (min … max):    8.220 s …  8.426 s    10 runs