A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Primary LanguageJavaScriptMIT LicenseMIT