internetarchive/dweb-mirror

Crawl and limitTotalTasks

Opened this issue · 0 comments

Either crawl is ignoring limitTotalTasks or its saying it completed when it didn't .

To test .... try a large crawl (like bali-lontar-transcribed) which is ~12k tasks
It will report that it got all of them, and appears to have done so, even though default limit is 3k tasks.