This middleware lets you use Scraper API for every request you process with Scrapy.
You could instead set the proxy field of your request's meta attribute to http://scraperapi.your=options:your_key@proxy-server.scraperapi.com:8001 and enable HttpProxyMiddleware.
This middleware's only purpose is to achieve this in a more convenient manner.
$ pip install scrapy-scraperapi-middleware
You need to specify your key for Scraper API in your settings.py or settings object.
SCRAPERAPI_KEY = 'your_key'You also need to enable ScrapyScraperAPIMiddleware as well as Scrapy's HttpProxyMiddleware.
DOWNLOADER_MIDDLEWARES = {
'scrapy_scraperapi_middleware.ScrapyScraperAPIMiddleware': 350,
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
}Scraper API supports additional options as documented here. You could use these options by adding them to your settings as dictionary.
SCRAPERAPI_OPTIONS = {
'render': 'true',
'country_code': 'us'
}If this package is helpful to you and you do not yet have a subscription to Scraper API, consider using my affiliate link if you plan on getting one. Be aware that all other links to ScraperAPI in this Readme are also affiliate links.