Wrong “fastest” labels

[RubenVerborgh]

A recent test gave me the following result:

• 132,803,732 – fastest
• 132,460,358
• 130,991,488 – fastest

while that should have been

• 132,803,732 – fastest
• 132,460,358 – fastest
• 130,991,488

[jdalton]

The margin of errors are what make the difference here. The ones highlighted are the fastest discounting their margin of errors.

[RubenVerborgh]

Thanks for explaining, but that doesn't explain everything I think. I subtracted the margin of errors:

• 131,223,367
• 130,632,405
• 129,838,762

So seems it should still be A & B.

[jdalton]

Hmm, I'll review that bit in the code. Would you share the jsperf test you saw this on.

[RubenVerborgh]

It was this one: http://jsperf.com/unknown-hash-key/2

[jdalton]

For some insight into our process we sort based on mean + moe but then run them through compare to perform a Mann-Whitney U test on them.

[jdalton]

If you can reproduce this again after your run would you open your web console and do ui.benchmarks.map(function(b){return JSON.stringify(b.stats,null,2)}) then paste the output into a gist and report back.

[RubenVerborgh]

Tried a few times but haven't been able to reproduce…
I suppose the above numbers and percentages are insufficient to make this into a unit test?

[jdalton]

Tried a few times but haven't been able to reproduce…

If you see it show up again then, no worries.

I suppose the above numbers and percentages are insufficient to make this into a unit test?

Right, we'd need the full stats record to be able to dig into it and see what's up.