jonasjacek/robots.txt
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
NOASSERTION
Stargazers
- abradabraAbraDabra
- aerophile@precog-iiitd
- andvgIT
- artisincursor|move
- chillmanschillmann.com
- chriseugenerodriguez@Microsoft
- demoklionPrague
- evanesotericShell Beach
- fabianski7
- floer32myxt.com
- igeligel@stripe
- JoshHarris85Tulsa
- kopaheadJohannesburg South Africa
- kringsjaaNorge
- lesh1kChisinau, Moldova
- lzfee0227China
- marcusrelacion
- MichelML@valence-labs
- mohagali
- NateMayChicago
- nmassiBuenos Aires
- nogweiiPhoenix, AZ
- paramdeoNew York
- pedrocasado
- Pheno-menal
- RogthatSinatraMillicodes
- shiv213University of Illinois at Urbana-Champaign
- sublocale
- Technetium1@garuda-linux @chaotic-aur
- thcolin@coopTilleuls
- thesubjectsteveIHS Markit
- tomcatfeveryoungstown, ohio
- vmlinz
- wyntonfranklinTrinidad and Tobago
- youtubetomp3converterapiVevioz Enterprise
- ZaeburnNew Zealand