Add rate limiting for site requests
afred opened this issue · 0 comments
afred commented
Because
Denying traffic based on info in the HTTP request and headers (IP range, user-agent, etc.) is reactionary (not preventive) and does not suffice in cases where there is no convenient way to specify an IP range and/or user-agent to target inconsiderate crawlers.
Background
Apache module mod_evasive can provide an effective speed limit for everybody and temporarily denying traffic (responding with 403) for those who exceed the speed limit.
This relieves us of having to track specific bot names or IP ranges.
Basic configuration of mod_evasive could limit requests to americanarchive.org to N visits per second... we just need to decide what the right value of N is.
Done when
- Requests to americanarchive.org are limited to a certain number of visits (tbd) per second.
- Rate limiting is part of provisioning new AAPB instances.