Does criterium focus on elapsed time measurements, ignoring sometimes-large GC time done in parallel on other CPU cores?

[jafingerhut]

I can do some further digging of my own into this question, but wanted to raise it here while thinking of it, having recently done some performance measurements using criterium on some single-threaded Clojure code, and noticing that the JVM process was fully utilizing 4 CPU cores on my multi-core laptop.

Here is what I am thinking:

Measuring elapsed times is very useful, and I would not want to eliminate such measurements in a tool like Criterium at all -- but perhaps supplementing those measurements with additional measurements on total CPU time used across all cores by the JVM, including parallel GC threads, and reporting those results separately, would have significant value, too.

Why? Because while sometimes you want to minimize latency of getting a result from a computation, but sometimes you may want to benchmark for total CPU time, because if you are renting CPU cores from a cloud service like AWS, it may be that total CPU time that you are spending money on, regardless of the start-to-finish time.

[jafingerhut]

It look like the main reason why running criterium to benchmark single-threaded code was taking 4 CPU cores on my system was that the JVM was configured to use a concurrent GC (by default with AdoptOpenJDK 8 and ZuluOpenJDK 11, it seems), and because System/gc was being called by criterium, even though I wasn't trying to ask it to do that, which might be considered a bug that would be fixed with this proposed change: #49

With that change, the System/gc calls performed by criterium no longer occur, and the CPU usage of that same single-threaded code benchmark goes down to 100% plus a few percent more, which seems more reasonable.

[jafingerhut]

Closing this issue, since PR #49 was merged and released as part of Criterium 0.4.6. Thanks!