web-scraping
There are 5137 repositories under web-scraping topic.
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
dgtlmoon/changedetection.io
The best and simplest free open source web page change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Evil0ctal/Douyin_TikTok_Download_API
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
lorien/awesome-web-scraping
List of libraries, tools and APIs for web scraping and data processing.
alirezamika/autoscraper
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
go-rod/rod
A Devtools driver for web automation and scraping
mherrmann/helium
Lighter web automation for Python
php-curl-class/php-curl-class
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
adbar/trafilatura
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
snooppr/snoop
Snoop — инструмент разведки на основе открытых данных (OSINT world)
lorien/grab
Web Scraping Framework
codingforentrepreneurs/30-Days-of-Python
Learn Python for the next 30 (or so) Days.
x4nth055/pythoncode-tutorials
The Python Code Tutorials
justmarkham/DAT8
General Assembly's 2015 Data Science course in Washington, DC
tidyverse/rvest
Simple web scraping for R
roach-php/core
The complete web scraping toolkit for PHP.
yifeikong/curl_cffi
Python binding for curl-impersonate via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
A9T9/RPA
UI.Vision: Open-Source RPA Software (formerly Kantu) - Modern Robotic Process Automation with Selenium IDE++
juancarlospaco/faster-than-requests
Faster requests on Python 3
rushter/selectolax
Python binding to Modest and Lexbor engines (fast HTML5 parser with CSS selectors).
vprusso/youtube_tutorials
Collection of scripts corresponding to LucidProgramming YouTube tutorials
intoli/user-agents
A JavaScript library for generating random user agents with data that's updated daily.
DataHenHQ/till
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
postmodern/spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
alecxe/scrapy-fake-useragent
Random User-Agent middleware based on fake-useragent
je-suis-tm/web-scraping
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
riimuru/gogoanime-api
Anime Streaming, Discovery API made with Cheerio and Express. Uses data from Gogoanime
dinubs/coolqlcool
Nextjs server to query websites with GraphQL
platonai/PulsarRPA
Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.
gosom/google-maps-scraper
scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
achuthasubhash/Complete-Life-Cycle-of-a-Data-Science-Project
Complete-Life-Cycle-of-a-Data-Science-Project
serpapi/google-search-results-python
Google Search Results via SERP API pip Python Package
programminghistorian/jekyll
Jekyll-based static site for The Programming Historian
AlexMathew/scrapple
A framework for creating semi-automatic web content extractors
spekulatius/PHPScraper
A universal web-util for PHP.