/simplerobotparser

A python script to fetch and parse robots.txt files for web-crawlers

Primary LanguagePython

Watchers