/search-bot-detector

Primary LanguageRubyApache License 2.0Apache-2.0

Search Bot Detector

Detect and verify select (i.e. Google and Yandex) search bots. Similar functionality is provided by the browser gem; this goes further in that we use DNS lookups to validate that a bot is actually what it says it is.

Supported Bots

  • Googlebot. Detected and verified as suggested here.
  • Yandex robot. Detected and verified as suggested here.

Installation

Simply add search-bot-detector to your Gemfile:

gem "search-bot-detector"

Usage

Super simple example:

require "search-bot-detector"

# IP address of `example.com` returns false.
SearchBotDetector.new("93.184.216.34", "Googlebot").any_bot?
# => false

# IP address of `crawl-66-249-66-1.googlebot.com` returns true.
SearchBotDetector.new("66.249.66.1", "Googlebot").any_bot?
# => true

# IP address of `spider-100-43-81-144.yandex.com` returns true.
SearchBotDetector.new("100.43.81.144", "+http://yandex.com/bots").any_bot?
# => true