scraping
There are 5698 repositories under scraping topic.
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
gocolly/colly
Elegant Scraper and Crawler Framework for Golang
psf/requests-html
Pythonic HTML Parsing for Humans™
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
code4craft/webmagic
A scalable web crawler framework for Java.
ultrafunkamsterdam/undetected-chromedriver
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
VinciGit00/Scrapegraph-ai
Python scraper based on AI
tabulapdf/tabula
Tabula is a tool for liberating data tables trapped inside PDF files
lorien/awesome-web-scraping
List of libraries, tools and APIs for web scraping and data processing.
alirezamika/autoscraper
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
MontFerret/ferret
Declarative web scraping
yujiosaka/headless-chrome-crawler
Distributed crawler powered by Headless Chrome
sparklemotion/mechanize
Mechanize is a ruby library that makes automated web interaction easy.
khuyentran1401/Data-science
Collection of useful data science topics along with articles, videos, and code
fake-useragent/fake-useragent
Up-to-date simple useragent faker with real world database
aapatre/Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
adbar/trafilatura
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
symfony/panther
A browser testing and web crawling library for PHP and Symfony
mendableai/firecrawl
🔥 Turn entire websites into LLM-ready markdown
snooppr/snoop
Snoop — инструмент разведки на основе открытых данных (OSINT world)
NikolaiT/GoogleScraper
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
geziyor/geziyor
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
emadehsan/thal
Getting started with Puppeteer and Chrome Headless for Web Scraping
lorien/grab
Web Scraping Framework
transitive-bullshit/awesome-puppeteer
A curated list of awesome puppeteer resources.
kevinzg/facebook-scraper
Scrape Facebook public pages without an API key
oscarotero/Embed
Get info from any web service or page
simonw/shot-scraper
A command-line utility for taking automated screenshots of websites
claffin/cloudproxy
Hide your scrapers IP behind the cloud. Provision proxy servers across different cloud providers to improve your scraping success.
lorey/mlscraper
🤖 Scrape data from HTML websites automatically by just providing examples
istresearch/scrapy-cluster
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
holgerd77/django-dynamic-scraper
Creating Scrapy scrapers via the Django admin interface
medialab/artoo
artoo.js - the client-side scraping companion.
scrapy/parsel
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
damklis/DataEngineeringProject
Example end to end data engineering project.
okfn-brasil/querido-diario
📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.