Robots.txt is a repository that contains 2 robots.txt file templates to help webmasters keep unwanted web robots (e.g. scraper bots, people search engines, seo tools, marketing tools, etc.) away from their websites but allow legitimate robots (e.g. search engine crawlers).
To be legitimate and get listed, robots must fully obey the Robots Exclusion Standard. The robots.txt file templates are a white list. Unlisted robots are, by the conventions of the Robots Exclusion Standard, not allowed to access.
The robots.txt template files contain an alphabetically ordered white list of legitimate web robots. In the commented version, each bot is shortly described in a comment above the (list of) user-agent(s). Uncomment or delete bots (User-agents) you do not wish to allow to access your website.
There are two robots.txt file versions:
- The regular file (with comments)
/robots.txt - The minified file (no comments)
/robots.min.txt
If you use the minified version, do not forget to rename it to robots.txt to be effective.
Compared to the vastness of the topic of blocking bots, robots.txt is a very small, private project! The author makes absolutely no claims and representations to warranties regarding the accuracy or completeness of the information provided. However, you can use the files in this repository AT YOUR OWN RISK.
The descision which bot is wanted/unwanted is done by the author, who is very conservative and opinionated when it comes to blocking bots. However, the author's decisions should be sufficient for many. Do not forget to adjust the list of allowed/forbidden directories to your needs.
Robots.txt by Jonas Jacek is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. Permissions beyond the scope of this license may be available upon request at http://jonas.me/.
Found a mistake? Want to help? Send a pull request!