[Feature] Add waybackmachine capability

[felipecaon]

In the past few days I've been experiencing using waybackmachine to enumerate robots.txt endpoints.

Sometimes robots.txt gets removed and sometimes the removed content can be juicy. Thus the ideia of searching every WBM to look for old robots entries.

I've implemented a quick and basic script to do a PoC, but I feel like this repo has the power to bring it to the next level since a lot of good features are already done.

https://gist.github.com/felipecaon/035ad1718c3cae681d2afb03c699795f

The gist works by getting all the robots.txt entries from WBM, parsing and sending to stdout. The script does not remove dps, just do a basic word removal.

If I have the time I may be able to open a PR. But if someone wants to takes it further, I would love to see that. The core waybackmachine endpoints to be used are on my gist file.