(bench) hangs on eval

Question

(bench) hangs on eval

Opened this issue 12 years ago · 5 comments

user=> (use 'criterium.core)
user=> (bench (eval '(+ 3 4)))

There it hangs.
OSX 10.6.8
Clojure 1.4.0

Answer 1 · 2012-10-16T22:56:08.000Z

In function (estimate-execution-count) the test to get out of the loop is never satisfied because the new-cl-state is always different. With eval compiling code I can see how the classloader state will keep changing. (The compilation state is equal at some point.)

(defn estimate-execution-count
...
(if (>= t period)
(= cl-state new-cl-state)
(= comp-state new-comp-state))

I realize that we're also benchmarking compilation time when we use eval, but it still can be useful for comparing choices. I'll see if I can come up with something.

Answer 2 · 2012-10-17T01:33:26.000Z

Armando,

Thanks for tracking that down! I hadn't considered use with eval. I'm not sure criterium is necessarily required for that, as JIT compilation presumably never kicks in. I guess the sample statistics are still useful though, so maybe being able to disable that convergence test is useful.

Answer 3 · 2012-10-17T03:21:08.000Z

I think you're right; eval won't let criterium do its work. For measuring compile time plus an interpreter run, (time) is probably enough.

Answer 4 · 2013-04-14T01:22:58.000Z

I just ran into this issue. It would be nice to get the sample statistics out.

Answer 5 · 2013-04-14T02:16:05.000Z

@devn, @begriffs criterium checks that the expression being tested has jit'd before running benchmarks. Using eval, that obviously can't happen. I guess we could add a flag to turn off that check, but you would still be timing compilation time + expression execution time.