retrying request after ban does not respect DOWNLOAD_DELAY and autothrottle
kakarukeys opened this issue · 0 comments
kakarukeys commented
It seems the retry request is immediately fired with a different proxy. In the log below you can see 6 requests fired within 1 second.
2019-11-30 00:00:53 [rotating_proxies.expire] DEBUG: Proxy <http://51.158.68.133:8811> is DEAD
2019-11-30 00:00:53 [rotating_proxies.middlewares] DEBUG: Retrying <GET https://xx/> with another proxy (failed 1 times, max retries: 5)
2019-11-30 00:00:53 [rotating_proxies.expire] DEBUG: GOOD proxy became DEAD: <http://167.172.140.184:3128>
2019-11-30 00:00:53 [rotating_proxies.middlewares] DEBUG: Retrying <GET https://xx/> with another proxy (failed 2 times, max retries: 5)
2019-11-30 00:00:53 [rotating_proxies.expire] DEBUG: GOOD proxy became DEAD: <http://178.128.85.255:3128>
2019-11-30 00:00:53 [rotating_proxies.middlewares] DEBUG: Retrying <GET https://xx/> with another proxy (failed 3 times, max retries: 5)
2019-11-30 00:00:53 [rotating_proxies.expire] DEBUG: GOOD proxy became DEAD: <http://186.219.183.45:8080>
2019-11-30 00:00:53 [rotating_proxies.middlewares] DEBUG: Retrying <GET https://xx/> with another proxy (failed 4 times, max retries: 5)
2019-11-30 00:00:53 [rotating_proxies.expire] DEBUG: GOOD proxy became DEAD: <http://103.123.246.66:8080>
2019-11-30 00:00:53 [rotating_proxies.middlewares] DEBUG: Retrying <GET https://xx/> with another proxy (failed 5 times, max retries: 5)
2019-11-30 00:00:53 [rotating_proxies.expire] DEBUG: Proxy <http://201.48.61.1:3128> is DEAD
2019-11-30 00:00:53 [rotating_proxies.middlewares] DEBUG: Gave up retrying <GET https://xx/> (failed 6 times with different proxies)
I could not exactly see how this is happening after examining your code, but from the answer below, it seems instead of returning a new copy of the request with return you should use yield?