scraping
There are 7111 repositories under scraping topic.
firecrawl/firecrawl
🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
feder-cr/Jobs_Applier_AI_Agent_AIHawk
AIHawk aims to easy job hunt process by automating the job application process. Utilizing artificial intelligence, it enables users to apply for multiple jobs in a tailored way.
gocolly/colly
Elegant Scraper and Crawler Framework for Golang
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
soxoj/maigret
🕵️♂️ Collect a dossier on a person by username from thousands of sites
psf/requests-html
Pythonic HTML Parsing for Humans™
ultrafunkamsterdam/undetected-chromedriver
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
code4craft/webmagic
A scalable web crawler framework for Java.
D4Vinci/Scrapling
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
lorien/awesome-web-scraping
List of libraries, tools and APIs for web scraping and data processing.
tabulapdf/tabula
Tabula is a tool for liberating data tables trapped inside PDF files
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
alirezamika/autoscraper
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
autoscrape-labs/pydoll
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
MontFerret/ferret
Declarative web scraping
yujiosaka/headless-chrome-crawler
Distributed crawler powered by Headless Chrome
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
jaypyles/Scraperr
Self-hosted webscraper.
sparklemotion/mechanize
Mechanize is a ruby library that makes automated web interaction easy.
CodeCutTech/Data-science
Collection of useful data science topics along with articles, videos, and code
fake-useragent/fake-useragent
Up-to-date simple useragent faker with real world database
daijro/camoufox
🦊 Anti-detect browser
bjesus/pipet
Swiss-army tool for scraping and extracting data from online assets, made for hackers
d60/twikit
Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot
snooppr/snoop
Snoop — инструмент разведки на основе открытых данных (OSINT world)
aapatre/Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
symfony/panther
A browser testing and web crawling library for PHP and Symfony
kevinzg/facebook-scraper
Scrape Facebook public pages without an API key
geziyor/geziyor
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
NikolaiT/GoogleScraper
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
transitive-bullshit/awesome-puppeteer
A curated list of awesome puppeteer resources.
lorien/grab
Web Scraping Framework
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
emadehsan/thal
Getting started with Puppeteer and Chrome Headless for Web Scraping