robots-parser
There are 26 repositories under robots-parser topic.
samclarke/robots-parser
NodeJS robots.txt parser with support for wildcard (*) matching.
scrapy/protego
A pure-Python robots.txt parser with support for modern conventions.
VIPnytt/RobotsTxtParser
An extensible robots.txt parser and client library, with full support for every directive and specification.
andreburgaud/robotspy
Alternative robots parser module for Python
messense/robotparser-rs
robots.txt parser for Rust.
fooock/robots.txt
:robot: robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API
samclarke/robotstxt
Go robots.txt parser
chrisakroyd/robots-txt-parser
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
ravern/gollum
Robots.txt parser and fetcher for Elixir
b4dnewz/robots-parse
A lightweight and simple robots.txt parser in node
eliasdabbas/robotstxt_app
Visual App for Testing URLs and User-agents blocked by robots.txt Files
larevanchedessites/google-robotstxt-ruby
🤖 Ruby gem wrapper around Google Robotstxt Parser C++ library
AntoineGagne/robots
A parser for robots.txt with support for wildcards. See also RFC 9309.
muratgozel/robotstxt-util
RFC 9309 spec compliant robots.txt builder and parser. 🦾 No dependencies, fully typed.
rimiti/robotstxt
Robots.txt parser and generator - Work in progress
mounicmadiraju/robot.txt-changes
💧 Test your robots.txt with this testing tool. Check if a URL is blocked, which statement is blocking it and for which user agent. You can also check if the resources for the page (CSS and JavaScript) are disallowed!. Robots.txt files help you guide how search engines crawl your site, and can be an integral part of your SEO strategy.
nicholasbergesen/robots-parser
Parse robots.txt and traverse sitemaps.
jwmorley73/jwm.robotstxt
Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.
ptsochantaris/can-proceed
A small, tested, no-frills parser of robots.txt files in Swift.
rimiti/robotizer
Robots.txt parser / generator
drmathias/robots
Parse robots.txt and sitemaps using dotnet
hgruniaux/robotstxt
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
menno120/robots-parser
Typescript robots.txt parser with support for wildcard (*) matching.
toimik/RobotsProtocol
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
0xIbra/robots-txt-component
Fully native robots.txt parsing component without any dependencies.
thomasleveil/pyrobots
python binding for Google robots.txt parser C++ library