TeamHG-Memex/scrapy-rotating-proxies

integrate middleware with standalone Scrapy script

beanben opened this issue · 1 comments

Hello hello,

I am running a standalone Scrapy script using the following settings:

DOWNLOADER_MIDDLEWARES = {
      'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
      'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
  }

  custom_settings = {
      "LOG_LEVEL": "INFO",
      "LOG_FORMAT": "%(levelname)s: %(message)s",
      "LOG_FILE": "rotation_test_log.txt",
      "ROTATING_PROXY_LOGSTATS_INTERVAL": 1,
      "DOWNLOADER_MIDDLEWARES": DOWNLOADER_MIDDLEWARES
  }

Looking at the log text, I can see that rotating_proxies.middlewares.BanDetectionMiddleware gets correctly passed but I seem to be missing the RotatingProxyMiddleware. See extract of log file below:

INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'rotating_proxies.middlewares.BanDetectionMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']

What am I missing ?

I was adding the list of proxies via an init method whereas I should add the list of proxies either via a link or the list itself directly in my custom settings