scraping-websites
There are 1991 repositories under scraping-websites topic.
MontFerret/ferret
Declarative web scraping
Anorov/cloudflare-scrape
A Python module to bypass Cloudflare's anti-bot page.
elixir-crawly/crawly
Crawly, a high-level web crawling & scraping framework for Elixir.
gildas-lormeau/single-file-cli
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
AmmeySaini/Edu-Mail-Generator
Generate Free Edu Mail(s) within minutes
josephlimtech/linkedin-profile-scraper-api
🕵️♂️ LinkedIn profile scraper returning structured profile data in JSON.
Python-World/Python_and_the_Web
Build Bots, Scrape a website or use an API to solve a problem.
slotix/dataflowkit
Extract structured data from web sites. Web sites scraping.
csbun/thal
译文：Puppeteer 与 Chrome Headless —— 从入门到爬虫
KTZgraph/sarenka
OSINT tool - gets data from services like shodan, censys etc. in one app
spekulatius/PHPScraper
A universal web-util for PHP.
avidLearnerInProgress/python-automation-scripts
Simple yet powerful automation stuffs.
oxylabs/quick-start-guide
Python quick start guides to get the most out of Oxylabs' Web Scraper API free trial.
baptisteArno/tinking
🧶 Extract data from any website without code, just clicks.
unixfox/pupflare
A webpage proxy that request through Chromium (puppeteer) - can be used to bypass Cloudflare anti bot / anti ddos on any application (like curl)
lkuffo/web-scraping
Más de 50 ejemplos de web scraping utilizando: Requests | Scrapy | Selenium | LXML | BeautifulSoup
crwlrsoft/crawler
Library for Rapid (Web) Crawler and Scraper Development
kennethreitz/requests-html
Pythonic HTML Parsing for Humans™
Go-phie/gophie
An Aggregator Engine for searching and downloading movies free - NO ADs!
driscoll42/ebayMarketAnalyzer
Scrape all eBay sold listings to determine average/median pricing, plot listings over time with trend lines, and extract to excel
m92vyas/llm-reader
Turn Webpage to LLM friendly input text. Similar to Firecrawl and Jina Reader API. Makes RAG, AI web scraping, image & webpage links extraction easy.
e43b/Kemono-and-Coomer-Downloader
The Kemono and Coomer Downloader simplifies downloading posts from Kemono and Coomer websites, allowing users to download individual or multiple posts, including entire profiles. It offers advanced features like downloading attachments, videos, and automatically organizing files.
RuthGnz/SpyScrap
CLI and GUI for OSINT. Are you very exhibited on the Internet? Check it! Twitter, Tinder, Facebook, Google, Yandex, BOE. It uses facial recognition to provide more accurate results.
DiegoCaraballo/Email-extractor
The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url
RyuzakiH/CloudflareSolverRe
Cloudflare Javascript & reCaptcha challenge (I'm Under Attack Mode or IUAM) solving / bypass .NET Standard library.
hridaydutta123/the-youtube-scraper
Download YouTube video description and video comments without using the YouTube API.
Bishalsarang/Leetcode-Questions-Scraper
Scrape Algorithm Questions from leetcode and generate html and epub file
johnbumgarner/newspaper3_usage_overview
This repository provides usage examples for the Python module Newspaper3k.
yousefkotp/Movies-and-Series-Scraper
A console application to scrape a valid watching links for any movie or series with exact season and episode number, you can also download a whole season with one click.
html2rss/html2rss
📰 Build RSS 2.0 feeds from websites (and JSON APIs) automatically or with a few CSS selectors.
alash3al/scraply
Scraply a simple dom scraper to fetch information from any html based website
autogram-is/spidergram
Structural analysis tools for complex web sites
pavlovtech/WebReaper
Web scraper, crawler and parser in C#. Designed as simple, declarative and scalable web scraping solution.
fedecalendino/nintendeals
Library with a set of tools for scraping information about Nintendo games and its prices across all regions (NA, EU and JP).
voliveirajr/seleniumcrawler
An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site
fernandod1/Instagram-to-discord
Monitor instagram user account and automatically post new images to discord channel via a webhook. Working 2022!