How to control concurrency?
luzihang123 opened this issue · 1 comments
luzihang123 commented
demand:10W web pages rendered into Base64 upload OSS one day
elacuesta commented
First of all, thanks for taking an interest in this project.
Concurrency should be handled by the Scrapy Scheduler (which acts before the handler provided by this package), taking into account settings like CONCURRENT_REQUESTS, CONCURRENT_REQUESTS_PER_DOMAIN and CONCURRENT_REQUESTS_PER_IP. Could you elaborate a bit more on you use case, maybe with some minimal code to help me understand the scenario? Thanks again.