Easy to use extension of Net::HTTP to let you obey robots.txt while crawling/scrapping/mining
Primary LanguageRuby