TeamHG-Memex/scrapy-rotating-proxies

What about getting proxy list from url?

aidiss opened this issue · 2 comments

It could work as something like this

ROTATING_PROXY_LIST_URL = 'http://..../get_proxies.json'

Or, what would be the correct way of implementing this?

  def getProxyList():
          resp = requests.get('https:/name/api/proxylist.txt')
          return resp.text.split()

  ROTATING_PROXY_LIST = getProxyList()

Same with json, you need to parse response of request as json

How to update proxy list every month, without stop the scrapper app ??