/WayRobots

Tool to find stored robots.txt files from the past

Primary LanguagePython

WayRobots

A tool to find snapshots of robots.txt file stored at http://web.archive.org

Usage

Before you use make sure to have the required python libraries

$ pip install requests
$ pip install colorama

Basic use

$ python wayrobots.py -i target.com -y 2015-2019

Help page

$ python wayrobots.py -h