crawler

There are 7392 repositories under crawler topic.

  • scrapy

    scrapy/scrapy

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    Language:Python58.2k1.8k3.2k11k

  • firecrawl/firecrawl

    The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data 🔥

    Language:TypeScript57.5k4.8k

  • NaiboWang/EasySpider

    A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

    Language:JavaScript42.5k2487215.2k

  • iawia002/lux

    👾 Fast and simple video download library and CLI tool written in Go

    Language:Go30.4k3861.1k3.2k

  • gocolly/colly

    Elegant Scraper and Crawler Framework for Golang

    Language:Go24.6k3275581.8k

  • jhao104/proxy_pool

    Python ProxyPool for web spider

    Language:Python22.8k4466205.3k

  • ScrapeGraphAI/Scrapegraph-ai

    Python scraper based on AI

    Language:Python21.3k1344061.8k
  • crawlee

    apify/crawlee

    Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

    Language:TypeScript19.5k1171k1k

  • binux/pyspider

    A Powerful Spider(Web Crawler) System in Python.

    Language:Python16.8k8918233.7k

  • codelucas/newspaper

    newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

    Language:HTML14.8k3836772.1k

  • shengqiangzhang/examples-of-web-crawlers

    一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

    Language:HTML14.4k3471143.8k
  • Douyin_TikTok_Download_API

    Evil0ctal/Douyin_TikTok_Download_API

    🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

    Language:Python14.3k815302.1k

  • projectdiscovery/katana

    A next-generation crawling and spidering framework.

    Language:Go14.2k93348788

  • s0md3v/Photon

    Incredibly fast crawler designed for OSINT.

    Language:Python11.9k3211101.6k
  • crawlab

    crawlab-team/crawlab

    Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

    Language:Go11.9k2139921.9k

  • code4craft/webmagic

    A scalable web crawler framework for Java.

    Language:Java11.6k7609464.2k

  • ssssssss-team/spider-flow

    新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

    Language:Java10.9k93432.1k

  • injetlee/Python

    Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

    Language:Python10.3k733624.3k

  • guyueyingmu/avbook

    AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

    Language:PHP9.8k3451392k

  • TeamWiseFlow/wiseflow

    Use LLMs to dig out what you care about from massive amounts of information and a variety of sources daily.

    Language:Python7.8k703141.4k

  • lorien/awesome-web-scraping

    List of libraries, tools and APIs for web scraping and data processing.

    Language:Makefile7.3k23210806
  • Scrapling

    D4Vinci/Scrapling

    🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

    Language:Python7.3k2621410

  • BruceDone/awesome-crawler

    A collection of awesome web crawler,spider in different languages

    7k20319730
  • autoscraper

    alirezamika/autoscraper

    A Smart, Automatic, Fast and Lightweight Web Scraper for Python

    Language:Python6.9k12465709

  • bda-research/node-crawler

    Web Crawler/Spider for NodeJS + server-side jQuery ;-)

    Language:TypeScript6.8k254306879

  • apify/crawlee-python

    Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

    Language:Python6.3k34373439

  • chyroc/WechatSogou

    基于搜狗微信搜索的微信公众号爬虫接口

    Language:Python6.1k2771911.7k
  • ferret

    MontFerret/ferret

    Declarative web scraping

    Language:Go5.9k99299309

  • rmax/scrapy-redis

    Redis-based components for Scrapy.

    Language:Python5.6k2711951.6k

  • yujiosaka/headless-chrome-crawler

    Distributed crawler powered by Headless Chrome

    Language:JavaScript5.6k115135409

  • SpiderClub/haipproxy

    :sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

    Language:Python5.5k20497909

  • autoscrape-labs/pydoll

    Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

    Language:Python5.2k2450285

  • DropsDevopsOrg/ECommerceCrawlers

    实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:

    Language:Python5.2k144341.4k

  • niespodd/browser-fingerprinting

    Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

    Language:JavaScript4.9k689262

  • adbar/trafilatura

    Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

    Language:Python4.7k31404288

  • myreader-io/myGPTReader

    A community-driven way to read and chat with AI bots - powered by chatGPT.

    Language:Python4.4k5135452