spatie/robots-txt
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
PHPMIT
Stargazers
- adamlacombe@edgeworkscreative @InteractiveTraining
- AllCdnBoyFitness Management
- AmazingVanishVanish's Vault
- balt7ed
- brendtBelgium
- briankiragu@Amprest-Technologies
- ChVuagniaux@inetis-ch
- codyphobeOregon, USA
- daanerFreelance
- freekmurze@spatie
- fynduck
- kavinsky@secretaligner
- lloricodeHalcyon Agile
- madeinnordesteCommunicatti
- maxmadknightPowerCode
- melihovvRussia, Volgograd
- michamilzSchwerin, Germany
- miclf
- MindiRogersGreenJupiter Games
- omeltchuk
- otienog1
- patrickbrouwersMaastricht
- peter279kInstitute for Information Industry
- pogorelschiBuckminster
- repatRepat LLC
- rodion1322
- ryanmortierOntario, Canada
- samwrigleyEcologi
- sarveshacharyaChandigarh, India
- sebastiandedeyne@spatie
- skyrpex@winglang
- tallcoderXpressive Technology LLC
- ThomasNoergaardWacky Studio IVS
- tn930
- tonysm@tighten
- TVkeDistribold