Purpose

This library will identify bot based on user agents. It does this using regex defined in src/main/resources/crawler_regex_list.txt.

This is based on/a port of Crawler-Detect.

Testing

Run sbt test.

Simply run sbt package.

sbt publishLocal

In build.sbt

libraryDependencies += "com.eventbrite" %% "scala-crawler-detect" % "1.0"

Using the isCrawler method will return a Boolean. Before checking this the user agent is cleaned up to prevent false positives.

import com.eventbrite.datafoundry.CrawlerDetect

val cd = new CrawlerDetect
val userAgent: String = ???
cd.isCrawler(userAgent)