Negative results for trivial benchmarks
jamii opened this issue · 3 comments
(with-progress-reporting (bench (let [x nil] (if x true false))))
Warming up for JIT optimisations 10000000000 ...
compilation occured before 2382422 iterations
classes loaded before 69087606 iterations
Estimating execution count ...
Sampling ...
Final GC...
Checking GC...
Finding outliers ...
Bootstrapping ...
Checking outlier significance
Evaluation count : 2439571620 in 60 samples of 40659527 calls.
Execution time mean : -1.293231 ns
Execution time std-deviation : 0.633958 ns
Execution time lower quantile : -2.010323 ns ( 2.5%)
Execution time upper quantile : 0.440421 ns (97.5%)
Overhead used : 26.202637 ns
Criterium estimates the function call overhead for wrapping the expression you pass it. The estimate is conservative, but can still be too high by the looks of things.
We should really warn if the measured time is less than some factor of the estimated function call overhead.
I worked around it by adding (bench (dotimes [_ n] ...)). Would it be reasonable to automatically add that when the estimated time is very low?
Sounds like a good idea.