robotstxt
There are 42 repositories under robotstxt topic.
dmi3kno/polite
Be nice on the web
alextim/astro-lib
Makes it easy to add robots.txt, sitemap and web app manifest during build to your Astro app.
Folyd/robotstxt
A native Rust port of Google's robots.txt parser and matcher C++ library.
itgalaxy/generate-robotstxt
Generator robots.txt for node js
ropensci/robotstxt
R 📦 for parsing and checking robots.txt files 🤖
itgalaxy/robotstxt-webpack-plugin
A webpack plugin to generate a robots.txt file
elchiconube/generate-robots-txt
The right robots.txt file for your project
andreburgaud/robotspy
Alternative robots parser module for Python
samclarke/robotstxt
Go robots.txt parser
datatxtorg/datatxt-spec
data.txt: Specification
aafeher/go-sitemap-parser
Go language library for parsing Sitemaps
herrbischoff/robots.txt
A sane, minimal robots.txt file (for the western world)
jasonmajors/robotstxt
This is a small package to make parsing robots.txt rules easier.
spcanelon/2022-ccd-sips
Workshop materials for creating the Center City Sips District 2022 interactive map
datatxtorg/datatxt-site
data.txt: Data for the Web
nicholasbergesen/robots-parser
Parse robots.txt and traverse sitemaps.
spire-rs/kit
📚 Libraries for retrieving and parsing robots.txt & sitemaps.
vxern/robots_txt
⚙️ A quality `robots.txt` ruleset parser to ensure your application follows the standard specification for the file.
ChrisWinters/robotstxt-manager
A Simple Robots.txt Manager Plugin For Wordpress.
littlebizzy/virtual-robotstxt
Replaces the default virtual robots.txt generated by WordPress with an editable one, and deletes any physical robots.txt file that may already exist.
MarcoDaniels/elm-robots-humans
Create robots.txt and humans.txt in Elm
MaximeGuinard/Robots.txt-Viewer
🌐 Displays the contents of robots.txt and sitemap.xml files of a website google extension
onuratakan/Site_Seo_Scanner
The site seo scanner.
rimiti/robotizer
Robots.txt parser / generator
s-thom/create-robots-txt-action
An action to create a robots.txt file from different sources
TahaT80/Robots_Scanner
Robots Scanner
Zer0Divis0r/rebuild-cache-sitemap
GitHub action to rebuild CDN cache according to sitemaps
ajayjsr/bloggerrobotstxtgenerator
The Blogger Robots.txt Generatois a tool designed to simplify the process of creating a robots.txt file for websites hosted on the Blogger platform. A robots.txt file is crucial for controlling how search engines index your site. This generator allows users to customize and generate a robots.txt file tailored to their specific needs.
ananthram-g/WebScraping
A Web Scraping project using robotstxt & rvest in R programming.
demyanovs/robotstxt
RobotsTXT parser reads and interprets robots.txt files into a struct.
jwmorley73/jwm.robotstxt
Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.
KnuckleheadsClub/rbdt
rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.
ossama131/Bias-to-Search-Engines-from-Robots.txt
Determining bias to search engines from Robots.txt
kritihq/robotstxt-generator
A simple HTML + JS web app that generate robots.txt file
peaceiris/docker-images
A collection of Docker images: robotstxt, linuxbrew, gcloud, and psql
prod3v3loper/php-honeypot-robots-bait
🍯 HoneyPot Robots Bait