robotshell/robotScraper
RobotScraper is a simple tool written in Python to check each of the paths found in the robots.txt file and what HTTP response code they return.
Python
RobotScraper is a simple tool written in Python to check each of the paths found in the robots.txt file and what HTTP response code they return.
Python