crawlers
There are 156 repositories under crawlers topic.
ai-robots-txt/ai.robots.txt
A list of AI agents and robots to block.
omrilotan/isbot
🤖/👨🦰 Detect bots/crawlers/spiders using the user agent string
flathunters/flathunter
A bot to help people with their rental real-estate search. 🏠🤖
salimk/Rcrawler
An R web crawler and scraper
StJudeWasHere/seonaut
Open source SEO auditing tool.
Norconex/crawlers
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
ArchiveTeam/wget-lua
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
narkhedesam/Proxy-List-Scrapper
Proxy List Scrapper
jonasjacek/robots.txt
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
behitek/social-scraper
Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)
howie6879/hproxy
hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)
joaopauloaramuni/python
Repo Python
Potelo/laravel-block-bots
Block crawlers and high traffic users on your site by IP using Redis
BaseMax/GooglePlayWebServiceAPI
Tiny script to crawl information of a specific application in the Google play/store base on PHP.
flulemon/sneakpeek
Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis
Symbolexe/Raven
Raven is a powerful and customizable web crawler written in Go.
peterbencze/serritor
Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.
herrbischoff/user-agents
User agent database in JSON format of bots, crawlers, certain malware, automated software, scripts and uncommon ones.
zcrawl/zcrawl
An open source web crawling platform
p0dalirius/crawlersuseragents
Python script to check if there is any differences in responses of an application when the request comes from a search engine's crawler.
anapaulagomes/licitacoes-de-feira
Licitações de Feira de Santana de fácil acesso aos cidadãos 🏦
delvelabs/htcap
htcap is a web application scanner able to crawl single page application (SPA) in a recursive manner by intercepting ajax calls and DOM changes.
robertciotoiu/mobile-de-car-data-collector
Crawl, scrape and persist Mobile.de car listings data in a smart & responsible way
ElektroStudios/Google-Search-URL-Crawler
Desktop app that crawls urls from Google's search engine results
arquejadalucy/jus_crawler
API que busca dados de um processo em todos os graus dos Tribunais de Justiça de Alagoas (TJAL) e do Ceará (TJCE).
BryanMorgan/isbot
Rust library to detect bots using a user-agent string
Hsins/Daily-GitHub-Trending
📰 Fetch daily trending repositories information on GitHub Trending Page by script writen in JavaScript and executed with GitHub Actions Service.
shaoxiongdu/SkyEye
一个基于SpringBoot的全网热点爬虫项目，原始热搜数据会入库，分词统计会存入Redis。方便之后的数据分析。
solidusio-contrib/solidus_sitemap
Provide a sitemap of your Solidus store.
tranlv/wiki-link
Scraping the wiki pages and find the minimum number of links between two wiki pages
BaseMax/StackoverflowCrawler
A web crawler which crawls the stackoverflow website.
versioneye/crawl_r
VersionEye crawlers implemented in Ruby.
romis2012/is-bot
Detect bots/crawlers/spiders via user-agent string
acidus99/Kennedy
Kennedy: Crawler and Search Engine for Gemini space. Leverages techniques and architecture from early WWW crawlers like Mercator, Archive.org, and GoogleBot
arthur3486/born2crawl
A highly performant and versatile crawling engine, designed with scalability and extensibility in mind.
FEZIRO/wechat-miniprogram-spider-demo
微信小程序云开发网络爬虫教程