crawler-commons/url-frontier

Set an explicit limit to the size of a key to 255 chars

jnioche opened this issue · 0 comments

This is the max length that a fully qualified domain name can have. Any URL sent to the Frontier with an explicit or computed key greater than the limit should be ignored.
This would prevent a situation where resources would get exhausted if very large keys were used