How to control the depth of crawling?
masterbo98 opened this issue · 1 comments
masterbo98 commented
When I user spidr, I find it really helpful and efficient. At present, I want to shorten the crawling time, is there any way or API to set the depth of spidr?
postmodern commented
You can use the max_depth: keyword argument which can be passed to Agent.start_at, .Agent.host, Agent.site, and Agent#initialize`.
https://rubydoc.info/gems/spidr/Spidr/Agent#initialize-instance_method