marcosbarbero/spring-cloud-zuul-ratelimit

High Performance degradation happen because of method synchronization

Mohamedfawzy1993 opened this issue · 3 comments

Mohamedfawzy1993 commented

Is your feature request related to a problem? Please describe.
I see that all consume methods are marked as synchronized, in case of "AbstractCacheRateLimiter" especially for Redis, synchronization in some cases "where the Redis server is slow or face networking issue" can cause Big performance issue,

here you can see the Performance degradation

Thread http-nio-8080-exec-40 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][487ms]
Thread http-nio-8080-exec-39 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][702ms]
Thread http-nio-8080-exec-38 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][873ms]
Thread http-nio-8080-exec-37 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][1082ms]
Thread http-nio-8080-exec-36 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][1281ms]
Thread http-nio-8080-exec-35 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][1494ms]
Thread http-nio-8080-exec-34 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][1695ms]
Thread http-nio-8080-exec-33 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][1901ms]
Thread http-nio-8080-exec-32 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][2073ms]
Thread http-nio-8080-exec-31 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][2275ms]
Thread http-nio-8080-exec-30 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][2486ms]
Thread http-nio-8080-exec-29 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][2687ms]
Thread http-nio-8080-exec-28 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][2895ms]
Thread http-nio-8080-exec-27 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][3106ms]
Thread http-nio-8080-exec-26 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][3283ms]
Thread http-nio-8080-exec-25 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][3495ms]
Thread http-nio-8080-exec-24 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][3650ms]
Thread http-nio-8080-exec-23 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][3816ms]
Thread http-nio-8080-exec-22 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][3962ms]
Thread http-nio-8080-exec-21 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][4114ms]
Thread http-nio-8080-exec-20 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][4289ms]
Thread http-nio-8080-exec-19 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][4451ms]
Thread http-nio-8080-exec-18 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][4609ms]
Thread http-nio-8080-exec-17 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][4768ms]
Thread http-nio-8080-exec-16 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][4929ms]
Thread http-nio-8080-exec-15 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][5088ms]
Thread http-nio-8080-exec-14 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][5246ms]
Thread http-nio-8080-exec-13 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][5406ms]
Thread http-nio-8080-exec-12 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][5577ms]
Thread http-nio-8080-exec-11 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][5742ms]
Thread http-nio-8080-exec-10 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][5899ms]
Thread http-nio-8080-exec-1 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][6044ms]
Thread http-nio-8080-exec-3 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][6177ms]
Thread http-nio-8080-exec-4 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][6311ms]
Thread http-nio-8080-exec-7 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][6445ms]
Thread http-nio-8080-exec-6 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][6577ms]
Thread http-nio-8080-exec-9 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][6710ms]
Thread http-nio-8080-exec-2 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][6844ms]
Thread http-nio-8080-exec-5 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][6977ms]

As you see the performance started from ~500ms and getting worse till 6 Seconds "Load test with 100 Thread & Redis server with latency +150ms"

The below is after removing Syncronization

Thread http-nio-8080-exec-25 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][728ms]
Thread http-nio-8080-exec-10 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][1011ms]
Thread http-nio-8080-exec-13 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][1009ms]
Thread http-nio-8080-exec-21 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][807ms]
Thread http-nio-8080-exec-20 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][852ms]
Thread http-nio-8080-exec-17 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][928ms]
Thread http-nio-8080-exec-37 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][423ms]
Thread http-nio-8080-exec-34 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][498ms]
Thread http-nio-8080-exec-33 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][523ms]
Thread http-nio-8080-exec-30 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][594ms]
Thread http-nio-8080-exec-27 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][649ms]
Thread http-nio-8080-exec-36 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][452ms]
Thread http-nio-8080-exec-10 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][983ms]
Thread http-nio-8080-exec-16 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][954ms]
Thread http-nio-8080-exec-39 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][481ms]
Thread http-nio-8080-exec-26 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][804ms]
Thread http-nio-8080-exec-31 RateLimiter Filter Response Time RateLimitPreFilter[SUCCESS][678ms]

Another point here, Redis is single-threaded, so it maintains synchronization by itself without any need to add extra synchronization "That is why the performance above may reach more than 1 second"

My suggestion here is to remove sync for Redis and depend on the Redis synchronization mechanism

Regards

marcosbarbero commented

Hi @Mohamedfawzy1993, thanks for filing this issue. Would it be possible for you to create a PR for that?