/scrapy-redis

Redis-based components for scrapy that allows distributed crawling. Small update to make it work for Scrapy 0.16+ and added QUEUE_TYPE and DUPE_FILTER options

Primary LanguagePythonBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

Watchers