grobotstxt is a native Go port of Google's robots.txt parser and matcher library.
Primary LanguageGoApache License 2.0Apache-2.0