p0dalirius/robotstester
This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.
PythonGPL-3.0
Issues
- 0
[Feature] Add waybackmachine capability
#1 opened by felipecaon
This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.
PythonGPL-3.0