TeamHG-Memex/scrapy-rotating-proxies

def response_is_ban. all the proxy in ban

kadimon opened this issue · 4 comments

I added a functions in my spider

def response_is_ban(self, request, response):
        return b'zzzz' not in response.body

    def exception_is_ban(self, request, exception):
        return None

After which all the proxies got the status of dead
I can not understand the problem

kmike commented

If b'zzzz' not in response.body is True (which is quite likely on a random page) then all responses are considered ban responses; it means scrapy-rotating-proxies thinks a proxy is banned and moves it to dead. If this is what happens then it works as intended.

"Metrika" is present on all pages of the site.

kmike commented

Could you please check what your response_is_ban method returns - e.g. print its result to console before returning it?

kmike commented

I'm closing it; feel free to reopen if there's some new information on how to debug this issue.