Spike of 524s from "/notices" API
Opened this issue · 1 comments
Problem
Earlier today on 8/11th/2022 we saw a spike of 524s from ClearlyDefined's "notice" endpoint between 6am and 6:40am PST. We didn't use to have this problem before, or at least it didn't last for that long, if was just a transient failure. CloudFlare returns 524s whenever a server doesn't reply before the connection times out. The timeout matches Cloudflare maximum timeout for non-Enterprise users which is 100 seconds. For Enterprise user it can be extended to 600 seconds, see doc.
Solution
ClearlyDefined
Increase Cloudfare timeout (if Cloudflare enterprise is being used) and/or improve ClearlyDefined service to be more performant.
Clients
Reduce batch size per request, we are currently sending 200 per request, we will decrease it to 100. Hopefully this doesn't introduce other type of throttling due to amount of requests.