Why is "Googlebot" not recognized without trailing slash?
egilburg opened this issue · 1 comments
egilburg commented
https://support.google.com/webmasters/answer/182072 says Googlebot
is used as user agent for the default web crawler.
But https://github.com/monperrus/crawler-user-agents/blob/master/crawler-user-agents.json#L3 only recognizes Googlebot/
with trailing slash.
Thanks.
monperrus commented
to differentiate the "main" Googlebot from Googlebot-Mobile, Googlebot-Image, Googlebot-Video, etc.