web-scraping
There are 6332 repositories under web-scraping topic.
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
dgtlmoon/changedetection.io
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Evil0ctal/Douyin_TikTok_Download_API
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
getmaxun/maxun
⚡ Easiest no code web data extraction platform • Instantly turn any website into API or spreadsheet ⚡
seleniumbase/SeleniumBase
Python APIs for web automation, testing, and bypassing bot-detection.
mherrmann/helium
Lighter web automation with Python
lorien/awesome-web-scraping
List of libraries, tools and APIs for web scraping and data processing.
D4Vinci/Scrapling
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
alirezamika/autoscraper
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
go-rod/rod
A Chrome DevTools Protocol driver for web automation and scraping.
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
snooppr/snoop
Snoop — инструмент разведки на основе открытых данных (OSINT world)
php-curl-class/php-curl-class
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
x4nth055/pythoncode-tutorials
The Python Code Tutorials
lorien/grab
Web Scraping Framework
mendableai/firecrawl-mcp-server
Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM clients.
gosom/google-maps-scraper
scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
codingforentrepreneurs/30-Days-of-Python
Learn Python for the next 30 (or so) Days.
oxylabs/amazon-scraper
Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.
A9T9/RPA
Ui.Vision Open-Source RPA Software with Computer Vision, OCR, Anthropic Computer Use/LLM. Selenium IDE import/export.
justmarkham/DAT8
General Assembly's 2015 Data Science course in Washington, DC
tidyverse/rvest
Simple web scraping for R
roach-php/core
The complete web scraping toolkit for PHP.
rushter/selectolax
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors.
juancarlospaco/faster-than-requests
Faster requests on Python 3
Smartproxy/Smartproxy
HTTP(S)/SOCKS5 rotating residential proxies - code examples & general information.
intoli/user-agents
A JavaScript library for generating random user agents with data that's updated daily.
gildas-lormeau/single-file-cli
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
vprusso/youtube_tutorials
Collection of scripts corresponding to LucidProgramming YouTube tutorials
kaliiiiiiiiii/Selenium-Driverless
a stealthy browser automation framework
je-suis-tm/web-scraping
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
DataHenHQ/till
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
postmodern/spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.