web-scraper
There are 916 repositories under web-scraper topic.
BruceDone/awesome-crawler
A collection of awesome web crawler,spider in different languages
php-curl-class/php-curl-class
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
arpit-omprakash/100ProjectsOfCode
A list of practical knowledge-building projects.
anaskhan96/soup
Web Scraper in Go, similar to BeautifulSoup
dipu-bd/lightnovel-crawler
Generate and download e-books from online sources.
juancarlospaco/faster-than-requests
Faster requests on Python 3
itsOwen/CyberScraper-2077
A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama
tholian-network/stealth
:rocket: Stealth - Secure, Peer-to-Peer, Private and Automateable Web Browser/Scraper/Proxy
spider-rs/spider
A web crawler and scraper for Rust
gosom/google-maps-scraper
scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
Oshan96/monkey-dl
Bulk download your favourite anime episodes from your favourite anime websites
postmodern/spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
k0rnh0li0/onlyfans-dl
OnlyFans content downloader
je-suis-tm/web-scraping
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
cassidoo/scrapers
A list of scrapers from around the web.
gildas-lormeau/single-file-cli
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
oxylabs/serp-scraper-api-guide
A quick-start guide on using SERP Scraper API
spekulatius/PHPScraper
A universal web-util for PHP.
AlexMathew/scrapple
A framework for creating semi-automatic web content extractors
austinoboyle/scrape-linkedin-selenium
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
shaikhsajid1111/social-media-profile-scrapers
Fetch user's data across social media
jaebradley/basketball_reference_web_scraper
NBA Stats API via Basketball Reference
oxylabs/how-to-scrape-google-scholar
A guide for extracting titles, authors, and citations from Google Scholar using Python and Oxylabs SERP Scraper API.
crwlrsoft/crawler
Library for Rapid (Web) Crawler and Scraper Development
oxylabs/web-unblocker
Free trial Web Unblocker - an AI-powered proxy solution that can bypass even the most sophisticated anti-bot systems.
paulpierre/markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
PhantomInsights/summarizer
A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.
duyet/awesome-web-scraper
A collection of awesome web scaper, crawler.
epiqueras/getsy
A simple browser/client-side web scraper.
passivebot/facebook-marketplace-scraper
This repository contains a script to scrape Facebook Marketplace data using Playwright, BeautifulSoup and Streamlit.
shaikhsajid1111/facebook_page_scraper
Scrapes facebook's pages front end with no limitations & provides a feature to turn data into structured JSON or CSV
lewisdonovan/google-news-scraper
Lightweight scraper for Google News
SenZmaKi/Senpwai
A desktop app for tracking and batch downloading anime
wikimedia/html-metadata
MetaData html scraper and parser for Node.js (supports Promises and callback style)
suntong/cascadia
Go cascadia package command line CSS selector
wearrrrr/HaiKei
HaiKei is an anime streaming website that uses the consumet API