Removing support for `Gocrawl` detection
summercms opened this issue · 0 comments
summercms commented
The user agent for Gocrawl
is as follows:
Googlebot (gocrawl v0.4)
From reading the closed issue found here: PuerkitoBio/gocrawl#41
The author has said the following: PuerkitoBio/gocrawl#41 (comment)
-
That package uses the name
Googlebot
in it's user agent but does notown
or haveany rights
to Google'scopyright
in using their name! -
This repo has been setup to detect fake bots and that user agent will be flagged now and will fail test runner.
-
The author even suggests for people to use their newer package called
fetchbot
which doesn't use the nameGoogleBot
-
Most firewall's block this
spoofing
attack.
Going to remove support for this third-party library as it is creating incorrect results and will not pass.