Test assignment
Obviously, there are lots of throttlers and load-balancers (nginx, HAProxy so on) in the wild, so I tried to not spend much time re-inventing it. Few years back, I already had to reinvent Guava as a test assignment: https://github.com/dk14/lifo-set.
Besides, developing and testing high quality throttler/balancer would take much more effort than appropriate for TA.
Notes:
- Just to have some fun, I decided to use scala3 (aka dotty) - so reading the code might feel a bit unusual (especially without "end" keyword).
- I think, I made few mistakes in the code/tests - nothing is perfect.
- Some parts of the TA were hinting at managing traffic with state-machines by using circuit breakers. Although it's easy to implement those - just dividing rps by 10 was good enough (there are nuances - see code comments).
- If I were to implement server app for this, I'd use something like Blaze for socket NIO + Gatling for SLA testing.
Assumptions:
- If you're not Facebook or Google - you won't have more than few mils of registered users
- with no scalability requirements - I assume it's even much less
- in any case, throttler can be scaled horizontally by userId if required
- I've sacrificed some neglibible GC time assuming there aren't too many unique users in 100ms window.
- I block threads during caching because I consider "cache update" overheads negligible
- Cache update tasks get fork-join pool (with unbounded queue) by default which is often dangerous but actually okay in here with hourly updates and explicit sla-service time-out. Note: Scala's
Await.result
has "blocking" annotation by default, so I didn't have to repeat it. - I don't block the throttling threads themselves of course. It's a bit nuanced - see comments about refresh vs evict in the code.
- Cache update tasks get fork-join pool (with unbounded queue) by default which is often dangerous but actually okay in here with hourly updates and explicit sla-service time-out. Note: Scala's
- If SlaService times-out - exception is propagated to Caffeine and staled value is not updated (according to their javadocs on CacheLoader)