TeamHG-Memex/scrapy-rotating-proxies

how to ban the proxy with timeout?

leolle opened this issue · 1 comments

leolle commented

I have the global DOWNLOAD_TIMEOUT in setting for 3 seconds, how to ban the proxy if timeout happens? Thank you.

PS, I can see the proxy with connection timeout is active all the time.

cpouldev commented

@leolle add this method to your spider

from twisted.internet.error import TimeoutError

def exception_is_ban(self, request, exception):
        return isinstance(exception, TimeoutError)