TeamHG-Memex/scrapy-rotating-proxies

Retry Request

Opened this issue · 1 comments

kadimon commented

Hello! Can you show me an example of a policy that will re-try to visit the page?

StasDeep commented

I know it's a late reply, but for future readers: the only thing a ban policy does is answering the two following questions:

  • "Are we banned, judging by the response?" (response_is_ban)
  • "Are we banned, judging be the exception?" (exception_is_ban).

It is then BanDetectionMiddleware's job to mark the response as banned and then it is RotatingProxyMiddleware's job to retry a request in case of ban.