How to use url_pattern ?
denapucaco opened this issue · 9 comments
I am trying to setup the rate limiting for few endpoints in a service where many endpoints are present.
As per the documentation url or url_pattern can be used to achieve this. But somehow rate limiting is not working, I have tried many combinations.
Can you please verify if the below configuration is correct?
ratelimit:
enabled: true
repository: JPA
policy-list: # Zuul rate limit policy list
audit-api: # custom policy name
- limit: 5 # request number limit per refresh interval window
refresh-interval: 60 # in seconds
type:
- user
- url_pattern=/api/v1.0.0/audit/*
I have tried with url as well. Replaced url_pattern with url in above config. But its not doing any rate limiting
- url=/api/v1.0.0/audit/statuscode/INP
If only url is used without value then rate limiting is working for all the urls, but we need it only for few.
We are using below versions
JDK = 11
spring-cloud-zuul-ratelimit = 2.4.2.RELEASE
Spring Boot = 2.2.7.RELEASE
Spring Cloud = Hoxton.SR4
Thanks in advance.
Prashant
Hello @denapucaco, thanks for getting in touch, we will get back to you asap! If you have issues in the 1.x.x.RELEASE line we recommend you to update to the latest version, unfortunately this line is not supported anymore.
The configuration you added to the description looks odd to me. Are you using it exactly the way you described? I'm talking about the full configuration not just the url_pattern
@marcosbarbero Thanks for your reply.
Below is the zuul config for service we are trying to add rate limiting. Please let me know if I am missing anything for url-pattern.
zuul:
routes:
audit-api:
path: /gw/audit/**
serviceId: audit-api-ms
stripPrefix: true
retryable: true
ratelimit:
enabled: true
repository: JPA
policy-list: # Zuul rate limit policy list
audit-api: # custom policy name
- limit: 2 # request number limit per refresh interval window
refresh-interval: 60 # in seconds
type:
- user
- url_pattern=/api/v1.0.0/audit/*
audit-api-ms:
ribbon:
listOfServers: ${SERVER_LIST_AUDIT}
ServerListRefreshInterval: 3600000
When I just use the url without value then rate table has below rows after I try few endpoints, so it rate limiting is working for all the urls if no url or url_pattern is mentioned in type. But we want it for just few urls
mi-gateway:audit-api:lossmgr:/api/v1.0.0/audit/statuscode/IPG -1 57011 2021-08-16 16:03:57
mi-gateway:audit-api:lossmgr:/api/v1.0.0/audit/statuscode/COM -1 57419 2021-08-16 16:04:06
mi-gateway:audit-api:lossmgr:/api/v1.0.0/audit/statuscode/WAT -1 30756 2021-08-16 16:03:22
Thanks you
I see, I'm not on my computer now but I suspect the serviceId being different than the uri is causing the problem.
Can you try to use audit-api-ms on the policy-list?
If it works then we have a bug 😬
Sure, I'll try and get back to you.
Thanks
Prashant
@marcosbarbero As per your suggestion tried with audit-api-ms in the policy-list. This didn't solve the issue.
I tried to debug the code to understand what could be causing the issue with url_pattern and found that -
com.marcosbarbero.cloud.autoconfigure.zuul.ratelimit.filters.AbstractRateLimitFilter Class ; policy method
policies = properties.getPolicies(routeId).stream()
.filter(policy -> applyPolicy(request, route, policy))
.collect(Collectors.toList());
When used the url_pattern the above code returns the empty list. I think because of this rate limiting is not working with url_pattern
I tried with url again and this time I could make it work with below config
ratelimit:
enabled: true
repository: JPA
policy-list: # Zuul rate limit policy list
audit-api: # custom policy name
- limit: 2 # request number limit per refresh interval window
refresh-interval: 60 # in seconds
type:
- user
- url=/api/v1.0.0/audit/statuscode/INP
- limit: 1 # request number limit per refresh interval window
refresh-interval: 60 # in seconds
type:
- user
- url=/api/v1.0.0/audit/statuscode/COM
- limit: 3 # request number limit per refresh interval window
refresh-interval: 60 # in seconds
type:
- user
- url=/api/v1.0.0/audit/statuscode/WAT
This created below rows in rate table. The URL is added twice in the rate_key. Is this a normal behavior?
mi-gateway:audit-api:lossmmgr:/api/v1.0.0/audit/statuscode/INP:/api/v1.0.0/audit/statuscode/INP -1 48423 2021-08-17 16:51:25
mi-gateway:audit-api:lossmmgr:/api/v1.0.0/audit/statuscode/COM:/api/v1.0.0/audit/statuscode/COM -1 43007 2021-08-17 16:51:42
mi-gateway:audit-api:lossmmgr:/api/v1.0.0/audit/statuscode/WAT:/api/v1.0.0/audit/statuscode/WAT -1 55993 2021-08-17 16:54:41
For now we want to use rate limiting with few URLs but we would like to make the url_pattern work. If you have any suggestions please let me know
Thanks,
Prashant
Thanks for the detailed response.
Can you try using ** instead of *?
I think I have tried it already but will try it one more time tomorrow and will update you.
@marcosbarbero Thank you for your help, I am able to make it work with url_pattern
I had to debug the code to understand the issue.
The request.getRequestURI() in the apply method of URL_PATTERN in RateLimitType class contains the URI with the path /gw/audit/. Since we are using the stripPrefix: true I was not adding this path in the url_pattern. ( url worked fine without this path )
Along with the path addition had to use ** as the AntPathMatcher uses them. I have added the config that worked for me below.
/**
* Rate limit policy considering an URL Pattern
*/
URL_PATTERN {
@Override
public boolean apply(HttpServletRequest request, Route route, RateLimitUtils rateLimitUtils, String matcher) {
return new AntPathMatcher().match(matcher.toLowerCase(), request.getRequestURI().toLowerCase());
}
Working Config
zuul:
routes:
audit-api:
path: /gw/audit/**
serviceId: audit-api-ms
stripPrefix: true
retryable: true
ratelimit:
enabled: true
repository: JPA
policy-list: # Zuul rate limit policy list
audit-api: # custom policy name
- limit: 2 # request number limit per refresh interval window
refresh-interval: 60 # in seconds
type:
- user
- url_pattern=/gw/audit/api/v1.0.0/audit/**
- http_method=get