Feature request: randomly sample from a parameter list
tavianator opened this issue · 1 comments
I am trying to write some production-grade benchmarks for bfs
with early termination, something like
bfs -name pattern -quit
where pattern
is the name of some file at a particular depth. However, I don't want to use the same file every time, because performance varies a lot depending on the order that files are visited in. A single file isn't representative.
This seems like a good case for -L
, like this
hyperfine -L name foo,bar,baz "bfs -name {name} -quit"
except that generates three different command lines and compares them against each other.
In real life I have a few dozen names, and I don't really care about the performance of each one separately. What I want is a mode that picks a random {name}
every time it runs the command, so I could do things like
hyperfine --parameter-sample name foo,bar,baz "bfs -name {name} -quit" "find -name {name} -quit"
and compare bfs
to find
over the whole set of names. Right now I'm hacking it with shuf
:
$ bfs -depth 5 -type f -printf '%f\n' | shuf -n20 >files
$ hyperfine -w1 'bfs -name $(shuf -n1 files) -quit' 'find -name $(shuf -n1 files) -quit'
Benchmark 1: bfs -name $(shuf -n1 files) -quit
Time (mean ± σ): 22.2 ms ± 12.9 ms [User: 17.5 ms, System: 98.0 ms]
Range (min … max): 0.9 ms … 53.2 ms 52 runs
Benchmark 2: find -name $(shuf -n1 files) -quit
Time (mean ± σ): 46.9 ms ± 35.1 ms [User: 15.2 ms, System: 31.4 ms]
Range (min … max): 5.2 ms … 107.3 ms 40 runs
Summary
bfs -name $(shuf -n1 files) -quit ran
2.11 ± 2.00 times faster than find -name $(shuf -n1 files) -quit
Thank you. Sounds like a useful feature. I would assume that most use cases for such an option would include numeric parameters, not a list of finite options. Do you think we could come up with a good CLI for this without having to add too many options?