How to do concurrent scraping?
vbuterin2 opened this issue · 2 comments
vbuterin2 commented
I currently use single thread scraper to crawl google.com. But I have tons of search terms. scrapy-selenium only open 1 browser, so I could only search one term at one time. Should I use remote browser or other solutions?
Flushot commented
@vbuterin2 I think your question is a duplicate of #76
cs246810 commented
You could run two scrapy crawl spider_name processes and set CONCURRENT_REQUESTS = 1.