A Distributed web crawler system. Support for templated spider development.
Primary LanguagePythonApache License 2.0Apache-2.0