A Java micro-benchmark example of Pattern find vs. matches using JMH framework
I needed to do micro-benchmarks on Java code and created this starter to help me in the future
mvn clean installSimply run using java -jar target/benchmarks.jar:
$ java -jar target/benchmarks.jar
# JMH version: 1.21
# VM version: JDK 11.0.1, OpenJDK 64-Bit Server VM, 11.0.1+13
# VM invoker: /Users/pelgi01/.sdkman/candidates/java/11.0.1-open/bin/java
# VM options: <none>
# Warmup: 2 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.sample.ArraySlicingBenchmark.testArrayStreamLimit
# Run progress: 0.00% complete, ETA 00:04:40
# Fork: 1 of 1
# Warmup Iteration 1: 1040818.801 ops/s
# Warmup Iteration 2: 1060593.136 ops/s
Iteration 1: 1066894.427 ops/s
Iteration 2: 1066709.854 ops/s
Iteration 3: 1066557.600 ops/s
Iteration 4: 1060663.407 ops/s
Iteration 5: 1069377.671 ops/s
Result "org.sample.ArraySlicingBenchmark.testArrayStreamLimit":
1066040.592 ±(99.9%) 12402.190 ops/s [Average]
(min, avg, max) = (1060663.407, 1066040.592, 1069377.671), stdev = 3220.809
CI (99.9%): [1053638.401, 1078442.782] (assumes normal distribution)
# JMH version: 1.21
# VM version: JDK 11.0.1, OpenJDK 64-Bit Server VM, 11.0.1+13
# VM invoker: /Users/pelgi01/.sdkman/candidates/java/11.0.1-open/bin/java
# VM options: <none>
# Warmup: 2 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.sample.ArraySlicingBenchmark.testArraysCopyRange
# Run progress: 25.00% complete, ETA 00:03:31
# Fork: 1 of 1
# Warmup Iteration 1: 26589243.469 ops/s
# Warmup Iteration 2: 28714683.597 ops/s
Iteration 1: 28909172.930 ops/s
Iteration 2: 28884393.950 ops/s
Iteration 3: 28839988.471 ops/s
Iteration 4: 28847179.788 ops/s
Iteration 5: 28696959.476 ops/s
Result "org.sample.ArraySlicingBenchmark.testArraysCopyRange":
28835538.923 ±(99.9%) 317347.449 ops/s [Average]
(min, avg, max) = (28696959.476, 28835538.923, 28909172.930), stdev = 82414.125
CI (99.9%): [28518191.474, 29152886.373] (assumes normal distribution)
# JMH version: 1.21
# VM version: JDK 11.0.1, OpenJDK 64-Bit Server VM, 11.0.1+13
# VM invoker: /Users/pelgi01/.sdkman/candidates/java/11.0.1-open/bin/java
# VM options: <none>
# Warmup: 2 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.sample.PatternMatchingBenchmark.testPatternFind
# Run progress: 50.00% complete, ETA 00:02:20
# Fork: 1 of 1
# Warmup Iteration 1: 650279.697 ops/s
# Warmup Iteration 2: 678440.017 ops/s
Iteration 1: 682620.764 ops/s
Iteration 2: 682781.705 ops/s
Iteration 3: 685321.757 ops/s
Iteration 4: 677914.475 ops/s
Iteration 5: 683559.497 ops/s
Result "org.sample.PatternMatchingBenchmark.testPatternFind":
682439.640 ±(99.9%) 10578.344 ops/s [Average]
(min, avg, max) = (677914.475, 682439.640, 685321.757), stdev = 2747.162
CI (99.9%): [671861.295, 693017.984] (assumes normal distribution)
# JMH version: 1.21
# VM version: JDK 11.0.1, OpenJDK 64-Bit Server VM, 11.0.1+13
# VM invoker: /Users/pelgi01/.sdkman/candidates/java/11.0.1-open/bin/java
# VM options: <none>
# Warmup: 2 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.sample.PatternMatchingBenchmark.testPatternMatch
# Run progress: 75.00% complete, ETA 00:01:10
# Fork: 1 of 1
# Warmup Iteration 1: 423777.701 ops/s
# Warmup Iteration 2: 450119.729 ops/s
Iteration 1: 447881.891 ops/s
Iteration 2: 450314.890 ops/s
Iteration 3: 388034.179 ops/s
Iteration 4: 405189.766 ops/s
Iteration 5: 411625.322 ops/s
Result "org.sample.PatternMatchingBenchmark.testPatternMatch":
420609.210 ±(99.9%) 105556.621 ops/s [Average]
(min, avg, max) = (388034.179, 420609.210, 450314.890), stdev = 27412.719
CI (99.9%): [315052.589, 526165.831] (assumes normal distribution)
# Run complete. Total time: 00:04:41
REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.
Benchmark Mode Cnt Score Error Units
ArraySlicingBenchmark.testArrayStreamLimit thrpt 5 1066040.592 ± 12402.190 ops/s
ArraySlicingBenchmark.testArraysCopyRange thrpt 5 28835538.923 ± 317347.449 ops/s
PatternMatchingBenchmark.testPatternFind thrpt 5 682439.640 ± 10578.344 ops/s
PatternMatchingBenchmark.testPatternMatch thrpt 5 420609.210 ± 105556.621 ops/sMIT © Gilad Peleg