/roboParser

A robots.txt parser that checks for response status and size for quick enumeration

Primary LanguagePythonGNU General Public License v3.0GPL-3.0

roboParser

A robots.txt parser that checks for response status and size for quick enumeration