TeamHG-Memex/scrapy-rotating-proxies

Auth Proxy

bezkos opened this issue · 5 comments

bezkos commented

I tried to use some auth proxies.
I had a file like
http://username:pass@url:port
......
but it dont seem to work.
Is it possible to use auth proxies with scrapy-rotating-proxies?

kmike commented

What Scrapy version are you using? Scrapy only added support for URL-based proxy auth in request.meta['proxy'] in 1.4 (see scrapy/scrapy#2530).

bezkos commented

I use Scrapy 1.2 and this addon atm for rotating proxies
https://github.com/aivarsk/scrapy-proxies
I think this part from scrapy-proxies
basic_auth = 'Basic ' + base64.b64encode(proxy_user_pass.encode()).decode()
request.headers['Proxy-Authorization'] = basic_auth
ll do auth proxies part.

kmike commented

Right, we may add a work-around for Scrapy < 1.4.

I'd prefer to rely on Scrapy to handle it though; is upgrading to 1.4 problematic? It should be backwards-compatible with 1.2, with a few minor exceptions.

bezkos commented

I think most of people use <1.4 especially in production environments(multi server need time to patch/test etc),.
So it'll be good to have such a feature for Scrapy <1.4.

Foxtrod89 commented

I'm using Scrapy 2.9.0 with Nord's SOCKS 5 proxies. Seems like scrapy-rotating-proxies can't verify my proxies listed in file.
2023-05-11 09:00:03 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 0, unchecked: 3, reanimated: 0, mean backoff time: 0s)
Can i have example of auth proxies stored in file?