/robots-txt-parser

This is an alternative which uses pre-fetched robots.txt files.

Primary LanguageJavaScriptMIT LicenseMIT

This is used for a private project.

This is an alternative which uses pre-fetched robots.txt files.