LoadTestRig vs JLBH vs JMH

[QIvan]

Hi! To continue my question on StackOverflow I'd like to ask about your new benchmarking harness.
What advantages do you see in LoadTestRing? For example, if we try to compare it with JLBH or JMH (maybe there are some other harnesses but I haven't seen any).

if it can win any other harness hands down, haven't you thought to make it a dedicated tool? (I find the world really needs good tools for macro benchmarks)

thank you!

[QIvan]

hi! @vyazelenko any thoughts? thanks.

[vyazelenko]

@QIvan We've built LoadTestRig for a very specific task, i.e. to do accurate latency measurements for Aeron. To that end the harness has two main features:

1. Send data at a given rate (e.g. 100K/sec) and record a full histogram of the results.
   It will detect if the rate cannot be achieved and warn about it. The harness also avoids Coordinated Omission problem by computing the send timestamp of when the message was suppose to be sent and thus will not hide outliers and bad cases.
2. Send bursts of messages according to the configuration (e.g. 10 msg/burst).
   In real systems messages don't arrive at a fixed rate of say 1 per ms, instead they are coming in bursts. What we have so far allows us to tests with static bursts (e.g. 100 msg/burst). Having bursts mean that the sends will have less often but more data/messages will be sent at a time. For example say the target rate is 1000 msg/sec, if the burst size is 1 (default) then a single message will be sent every millisecond. However if the burst size is 10 then a batch of 10 messages will be sent every 10th millisecond.
   What would be even more useful is to have support for dynamic bursts of random size and duration but this is some work for the future.

With JMH you cannot control the rate of calls. What you have is just the duration of a single iteration but you cannot control how many times the benchmark method will be invoked. There is also no mode which can record an entire histogram of the results. For example the org.openjdk.jmh.annotations.Mode#SampleTime will only sample the results but not record all of them. There is no support for batching out of the box either.

The JLBH seems much closer in its design and goals to LoadTestRig. I haven't used it myself to say much about it but from what I've seen while looking at your SO question is that it does execute calls at a rate which was somehow deduced from the number of calls you wanted to make and the default timeout of 5 seconds and that it handle the CO problem as well. Whether it records a full histogram of results or not or if it can do the burst of calls is not clear.