Use min/max from series to show error range
Closed this issue · 2 comments
It seems that JMH reports error range too broadly, and on plotter chart we can see a black line that crosses whole bar from left to right margin: https://screenshots.firefox.com/yttrsDp8HUoPP289/plokhotnyuk.github.io
IMHO instead that provided value we can use min/max values of data series.
Ya, i'm suffering this behavior too...
Think the thing is that JMH uses 99th percentile to calculate the error-score. And that can lead to a high error if there are few iterations, even if the iterations are yielding the same scores and haven't much derivation. So one remedy would probably to increase forks and iterations... though i'm refrain from that usually because if the results are stable with less iterations/forks i'm trying to safe execution time by not increasing the iterations/forks.
I was a bit hesitant so far to calculate my 'own' error instead of using the 'official' JMH score metric...
However, since we have the original iterations scores we could use min/max. Or we could calculate StdDev and use this...
Thoughts ?
cc @mgledi
Just inferred min/max from the raw data and used this instead of the confidence interval for displaying the error bar. Like it!