open source, distributed, restful crawler engine
this is a project base on scrapy,we reuse most of scrapy code in crawler part
I design my distributed architecture by imitating elasticsearch. it spire me to do a engine for distributed crawler
- cluster
- node
- webservice
- schedule
- crawl
- engine
you can see it in issues
it will be in wiki. so please look forward to it