TeamHG-Memex/scrapy-rotating-proxies

Fix the setting of `Proxy-Authorization` header

tianhuil opened this issue · 0 comments

Currently, the Proxy-Authorization header is not properly reset in between retries. This is because Proxy-Authorization is only set if the header is empty, which it is not as it was copied from a previous request. The fix is to always reset Proxy-Authorization when a new proxy is issued.

Specifically, line 153 of middlewares.py should be changed from

if creds and not request.headers.get('Proxy-Authorization'):

to

if creds:

@kmike: any thoughs?