Add AI agents
httpjamesm opened this issue · 1 comments
httpjamesm commented
AI tools like ChatGPT can crawl webpages. Their user agents should be added to this list.
https://darkvisitors.com is creating an index of AI crawler user agents.
monperrus commented
yes, great idea!