/PyCrawler

A highly modularized, easy config and scale python crawler

Primary LanguagePythonBSD 2-Clause "Simplified" LicenseBSD-2-Clause

PyCrawler

An easy config and scale python crawler

lxml dependencies: sudo apt-get install libxml2-dev libxslt-dev python-dev

cyclic complexity python pygenie.py complexity --verbose *py

test coverage nosetests -v --with-coverage --cover-package=Store --with-doctest --cover-erase --exe dependency: pip install coverage