Datenschule/jedeschule-scraper

Add Scrapy option to non-scrapy crawlers?

cyroxx opened this issue · 4 comments

cyroxx commented

For the sake of an experiment, I converted my WIP crawler for Hessen into a Scrapy spider: cyroxx/jedeschule-scraper@0263e41

Is this an approach that we should consider for those crawlers that are not yet scrapy based - at least as a second option? This would allow us to implement scrapy pipelines with them and use them on ScrapingHub.

k-nut commented

I think for the sake of having all the scrapers run through the pipelines and on scraping hub this is a really good idea!

knutator2 commented

can we close this one? @cyroxx @k-nut

k-nut commented

Yes, I think we have an agreement that this is a good idea ;)