Explore robots.txt files and visit the disallowed entries, recording the results and showing juicy URLs
Primary LanguageRuby