/os-scrapy-httpproxy-middleware

Primary LanguagePythonMIT LicenseMIT

os-scrapy-httpproxy-middleware

Build Status codecov PyPI - Python Version PyPI

This project provide middleware to enhance Scrapy built-in HttpProxyMiddleware. You can set proxy without sheme as the following format:

[user:password@]proxy:port

Install

pip install os-scrapy-httpproxy-middleware

You can run example spider directly in the project root path

scrapy crawl example

Usage

Settings

  • enable middleware, it is better disable Scrapy built-in HttpProxyMiddleware

    DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": None,
    "os_scrapy_httpproxy_middleware.middlewares.HttpProxyMiddleware": 750,
}

Unit Tests

sh scripts/test.sh

License

MIT licensed.