Predictable mode
bmeurer opened this issue · 6 comments
We need a predictable mode for the benchmark where each test is run for a fixed number of iterations. Currently benchmark.js doesn't seem to provide this option.
As @tebbi pointed out yesterday, we probably want to always run in this predictable mode, as with the variable iteration count and no separate warmup, we otherwise have variable influence of the first iteration on the overall score. Especially significant when running on fast vs slow machines.
We probably need to replace benchmark.js with something simpler that does just that for us.
cc @jdalton
How did this go from "Sure, we'd love to help" from our end (benchmark.js) to "We probably need to replace Benchmark.js" on your end (web-tooling-benchmark)? Last time I chatted with you, @bmeurer, you said you were pinging @mathiasbynens.
Fixed iterations aren't super great as you get into the SunSpider problem of near 0ms times. That said, as I said before I'm happy to add the option. Is another possible solution to have benchmark.js perform a warmup before calculating its run time?
+1 to not blindly switching to a custom solution — let’s evaluate extending Benchmark.js first. It sounds like @jdalton is open to this which is 👍 🎉 🎈♥
Fixed iterations aren't super great as you get into the SunSpider problem of near 0ms times. That said, as I said before I'm happy to add the option. Is another possible solution to have benchmark.js perform a warmup before calculating its run time?
@jdalton It sounds like both options could be useful in their own way, and could both be enabled in web-tooling-benchmark’s case specifically.
Sorry, I was not advocating against benchmark.js in general. Just noticed that there's not a lot of benefits to it, since we don't need the non predictable mode. @tebbi suggested to set maxTime to 0 and minSamples to the fixed iteration count as first step mitigation. I guess I'll try that.
I'd like to nevertheless include the first iteration to also measure that. So the only option is to have fixed iteration count.
So, what's your opinion on the work-around? Do you think we can have that feature soonish in general in benchmark.js?
So, what's your opinion on the work-around?
If it works, then rock 🤘 ! I'd want to dig into, and properly codify, the trick on our end though just to make it more discoverable (easier to consume).