TeamHG-Memex/scrapy-rotating-proxies

Make backoff cap configurable

petermoore14 opened this issue · 2 comments

The current backoff algorithm is quite effective, but I think it would benefit from supporting configuration of the 'cap' used for the time limit. It's very possible a cluster of proxies could go down for several days, but the fixed cap used at present ensures they'll be checked around once every half hour at minimum, which isn't terribly sensible in this use case.

The fix would be to simply support the ROTATING_PROXY_BACKOFF_CAP setting, which would be used in the same way as the existing ROTATING_PROXY_BACKOFF_BASE to initialize the backoff function.

kmike commented

Yeah, a good suggestion. +1 to add this option.

kmike commented

Fixed by #10 - thanks @petermoore14!