Distributed crawler that takes into consideration robots.txt, throttling and more.
Primary LanguageScala
This repository is not active