uber/RxDogTag

Create a representative performance benchmark

Closed this issue · 4 comments

Anecdotal evidence from uber internal is ok, but it would be nicer to have a benchmark project we could point. Maybe we could reuse RxJava's jmh tests, or the classic reactive streams shakespeare/scrabble benchmark?

Other considerations:

  • jmh/android benchmark/both?

Notes from talking with @dlew

  • RxJava/scrabble benchmarks might not be good options since they're focused on stream throughput, we care mostly about overhead of just subscribe points
  • Should have both simple just(1).subscribe() example as well as one where the stream is kept open

Other notes from myself

  • MemoryPerf from rxjava still probably relevant, can adapt to the above focus areas

Open question - still maybe worth doing the full rx tests just for more "real world" examples to know if the cost is effectively amortized

FWIW, I wrote some caliper benchmarks for a lib that solves the same problem to compare against another lib. Should be able to modify those tests to run against dogtags with a couple lines of code if you wanted.

https://github.com/halfhp/rxtracer/blob/master/rxtracer/src/test/java/com/halfhp/rxtracer/RxTracerBenchmark.java

I think the concerns I had when I was originally writing the tests were similar to yours. There are a few notes about the benchmarks in the readme if you're interested.

Thanks! We might give that a shot. Fortunately this is pretty targeted in the sense that it only happens at throughput (guarded delegate calls) and subscribe-time.

Closing this out since we have good benchmarks, documentation and a nice little script that outputs with the format we're looking for.