An App for testing URLs and User-agents blocked by robots.txt Files.
Given a certain robots.txt file, check whether or not any of the provided URLs can be fetched by all the user-agents in the robots.txt file (checks all the combinations).
eliasdabbas/robotstxt_app
Visual App for Testing URLs and User-agents blocked by robots.txt Files
Python