CrawlerDetect
CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent. Currently able to detect over 390 bots/spiders/crawlers.
Installation
Run composer require jaybizzle/crawler-detect 1.*
or add "jaybizzle/crawler-detect" :"1.*"
to your composer.json
.
Usage
use Jaybizzle\CrawlerDetect\CrawlerDetect;
$CrawlerDetect = new CrawlerDetect;
// Check the user agent of the current 'visitor'
if($CrawlerDetect->isCrawler()) {
// true if crawler user agent detected
}
// Pass a user agent as a string
if($CrawlerDetect->isCrawler('Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)')) {
// true if crawler user agent detected
}
// Output which regex rules matched (if any)
var_dump($CrawlerDetect->getMatches());
Contributing
If you find a bot/spider/crawler user agent that CrawlerDetect fails to detect, please submit a pull request with with the regex pattern added to the $crawlers
array in CrawlerDetect.php
and add the failing user agent to tests/crawlers.txt
.
Failing that, just create an issue with the user agent you have found, and we'll take it from there :)
Laravel Package
If you would like to use this with Laravel 5, please see Laravel-Crawler-Detect
Changelog
v1.0.9
- Added '007ac9 Crawler'
- Added 'Airmail'
- Added 'Anemone'
- Added 'Butterfly'
- Added 'Content Crawler'
- Added 'Digg'
- Added 'DomainAppender'
- Added 'EasouSpider'
- Added 'ElectricMonk'
- Added 'InAGist'
- Added 'IODC'
- Added 'iZSearch'
- Added 'FRCrawler'
- Added LinksCrawler
- Added 'Lipperhey Link Explorer'
- Added 'ltx71'
- Added 'MetaURI'
- Added 'MSIECrawler'
- Added 'ocrawler'
- Added 'Online Website Link Checker'
- Added 'OpenWebSpider'
- Added 'ow.ly'
- Added 'PercolateCrawler'
- Added 'Robosourcer'
- Added 'Scrapy'
- Added 'SpiderMan'
- Added 'SSL-Crawler'
- Added 'UnwindFetchor'
- Added 'urlresolver'
- Added 'XML Sitemaps Generator'
- Added 'Y!J-ASR'
- Added 'YisouSpider'
v1.0.8
- Added 'bl.uk_lddc_bot'
- Added 'classbot'
- Added 'CoPubbot'
- Added 'Domain Re-Animator Bot'
- Added 'Healthbot'
- Added 'IstellaBot'
- Added 'LinkpadBot'
- Added 'lufsbot'
- Added 'PaperLiBot'
- Added 'Plukkie'
- Added 'SearchmetricsBot'
- Added 'TrueBot'
- Added 'UnisterBot'
- Added 'YioopBot'
- Added 'Insitesbot'
- Added 'xintellibot'
- Added 'NerdyBot'
- Added 'NextGenSearchBot'
- Added 'ScreenerBot'
- Added 'ShowyouBot'
v1.0.7
- Added 'SurdotlyBot'
- Added 'AddThis'
- Tweaked some regex patterns
- Fixed #10
v1.0.6
- Added 'findxbot'
- Added 'SemrushBot'
- Added 'yoozBot'
v1.0.5
- Added 'GigablastOpenSource'
- Added 'MegaIndex.ru'
- Added 'Pingdom.com_bot'
- Added 'WeSEE:Ads/PageBot'
v1.0.4
- Added 'CrawlBot'
- Added 'Flamingo_SearchEngine'
- Added 'python-requests'
- Added 'Seznam screenshot-generator'
- Added 'SklikBot'
- Added 'trendictionbot'
v1.0.3
- Added 'BUbiNG'
- Added 'Qwantify'
- Added 'archive.org_bot'
- Added 'Applebot'
- Added 'TweetmemeBot'
v1.0.2
- Added 'AbiLogicBot'
- Added 'Link Valet'
- Added 'Mrcgiguy'
- Added 'LinkExaminer'
- Added 'LinksManager.com_bot'
- Added 'Notifixious'
- Added 'online link validator'
- Added 'Ploetz + Zeller'
- Added 'InfoWizards Reciprocal Link System PRO'
- Added REL Link Checker
- Added 'SiteBar'
- Added 'Vivante Link Checker'
v1.0.1
- Added 'Yahoo Link Preview' bot
Parts of this class are based on the brilliant MobileDetect