mr-n30/waybackrobots

Fetch a given sites robots.txt file from https://web.archive.org/

waybackrobots

Fetch a given sites robots.txt file from wayback machine

Installation:

git clone https://github.com/mr-n30/waybackrobots.git
cd waybackrobots/
pip install -r requirements.txt
python waybackrobots.py -h

Usage:

$ python waybackrobots.py -t 10 -o output.txt example.com