JayBizzle/Crawler-Detect

Google Ads bot to add

Opened this issue · 0 comments

Hello,

I did not see any rule that would detect following:

  • AdsBot-Google (+http://www.google.com/adsbot.html)
  • Google-Adwords-Instant (+http://www.google.com/adsbot.html)

Seems Google tends to create more and more robots with names.

Also looking at other existing headers. Google ads urls to their bots
+http://www.google.com/adsbot.html or +https://support.google.com/webmasters/answer/1061943

Maybe it would be a good idea to regex for google domain url?