robotshell/robotScraper
RobotScraper is a simple tool written in Python to check each of the paths found in the robots.txt file and what HTTP response code they return.
Python
No issues in this repository yet.
RobotScraper is a simple tool written in Python to check each of the paths found in the robots.txt file and what HTTP response code they return.
Python
No issues in this repository yet.