robots-exclusion-protocol
There are 14 repositories under robots-exclusion-protocol topic.
jimsmart/grobotstxt
grobotstxt is a native Go port of Google's robots.txt parser and matcher library.
cicirello/generate-sitemap
Generate an XML sitemap for a GitHub Pages site using GitHub Actions
bnomei/kirby3-robots-txt
Manage the robots.txt from the Kirby config file
andreburgaud/robotspy
Alternative robots parser module for Python
crwlrsoft/robots-txt
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
hrbrmstr/spiderbar
Lightweight R wrapper around rep-cpp for robot.txt (Robots Exclusion Protocol) parsing and path testing in R
bnomei/kirby-robots-writer
Robots for Kirby CMS
muratgozel/robotstxt-util
RFC 9309 spec compliant robots.txt builder and parser. š¦¾ No dependencies, fully typed.
VIPnytt/UserAgentParser
User-Agent parser for robots.txt, X-Robots-tag and Robots-meta-tag rule sets
drmathias/robots
Parse robots.txt and sitemaps using dotnet
hgruniaux/robotstxt
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
menno120/robots-parser
Typescript robots.txt parser with support for wildcard (*) matching.
toimik/RobotsProtocol
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
thomasleveil/pyrobots
python binding for Google robots.txt parser C++ library