Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls
Primary LanguagePython