webscraping
There are 9841 repositories under webscraping topic.
firecrawl/firecrawl
🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data
huginn/huginn
Create agents that monitor and act on your behalf. Your agents are standing by!
assafelovic/gpt-researcher
An LLM agent that conducts deep research (local and web) on any given topic and generates a long report with citations.
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
getmaxun/maxun
⚡ Easiest no code web data extraction platform • Instantly turn any website into API or spreadsheet ⚡
pystardust/ani-cli
A cli tool to browse and play anime
D4Vinci/Scrapling
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
lorien/awesome-web-scraping
List of libraries, tools and APIs for web scraping and data processing.
alirezamika/autoscraper
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
niespodd/browser-fingerprinting
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️♂️ when scraping the web?
jaypyles/Scraperr
Self-hosted webscraper.
daijro/camoufox
🦊 Anti-detect browser
scrapoxy/scrapoxy
Scrapoxy is a super proxies manager that orchestrates all your proxies into one place, rather than spreading management across multiple scrapers. It manages IP rotation and fingerprinting, and smartly routes traffic to avoid bans.
anaskhan96/soup
Web Scraper in Go, similar to BeautifulSoup
itsOwen/CyberScraper-2077
A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama
Kaliiiiiiiiii-Vinyzu/patchright
Undetected version of the Playwright testing and automation library.
reworkd/tarsier
Vision utilities for web interaction agents 👀
TheWebScrapingClub/webscraping-from-0-to-hero
The web scraping open project repository aims to share knowledge and experiences about web scraping with Python
requests-cache/requests-cache
Persistent HTTP cache for python requests
jamesturk/scrapeghost
👻 Experimental library for scraping websites using OpenAI's GPT API.
m8sec/CrossLinked
LinkedIn enumeration tool to extract valid employee names from an organization through search engine scraping
raznem/parsera
Lightweight library for scraping web-sites with LLMs
holgerd77/django-dynamic-scraper
Creating Scrapy scrapers via the Django admin interface
mov-cli/mov-cli
Watch everything from your terminal.
GodsScion/Auto_job_applier_linkedIn
Make your job hunt easy by automating your application process with this Auto Applier
Kaliiiiiiiiii-Vinyzu/patchright-python
Undetected Python version of the Playwright testing and automation library.
cdpdriver/zendriver
A blazing fast, async-first, undetectable webscraping/web automation framework based on ultrafunkamsterdam/nodriver. Now with Docker support!
benibela/xidel
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Skallwar/suckit
Suck the InTernet
maxhumber/gazpacho
🥫 The simple, fast, and modern web scraping library
scrapfly/scrapfly-scrapers
Scalable Python web scraping scripts for +40 popular domains
z0m31en7/Uscrapper
Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.
wodsuz/EasyApplyJobsBot
A python bot to automatically apply all Linkedin,Glassdoor, etc Easy Apply jobs based on your preferences. Auto login, auto fill additional questions, apply automatically!
chris-greening/instascrape
Powerful and flexible Instagram scraping library for Python, providing easy-to-use and expressive tools for accessing data programmatically
openzim/zimit
Make a ZIM file from any Web site and surf offline!
vil/H4X-Tools
Open source toolkit for scraping, OSINT and more.