hasmanydevelopers/robot_rules
A tool to determine if the robots.txt would prevent a given user agent from making a request to a given URI.
RubyMIT
Stargazers
No one’s star this repository yet.
A tool to determine if the robots.txt would prevent a given user agent from making a request to a given URI.
RubyMIT
No one’s star this repository yet.