Google Ads bot to add
Opened this issue · 0 comments
gregzawadzki commented
Hello,
I did not see any rule that would detect following:
AdsBot-Google (+http://www.google.com/adsbot.html)
Google-Adwords-Instant (+http://www.google.com/adsbot.html)
Seems Google tends to create more and more robots with names.
Also looking at other existing headers. Google ads urls to their bots
+http://www.google.com/adsbot.html
or +https://support.google.com/webmasters/answer/1061943
Maybe it would be a good idea to regex for google domain url?