TeamHG-Memex/scrapy-rotating-proxies

Insensible middleware priority recommendations

Opened this issue · 2 comments

3hhh commented

In your README you recommend setting

    'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
    'rotating_proxies.middlewares.BanDetectionMiddleware': 620,

This however is insensible as the default settings have 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590, which means that a custom response_is_ban(self, request, response) implementation receives a gzipped response and response.css('whatever') leads to exceptions.

To avoid that, something like

    'rotating_proxies.middlewares.RotatingProxyMiddleware': 582,
    'rotating_proxies.middlewares.BanDetectionMiddleware': 583,

makes more sense.

have the same issue
@3hhh does changing original order break any functionality?

3hhh commented

None that I'm aware of.