webscraper
There are 1507 repositories under webscraper topic.
jaypyles/Scraperr
Self-hosted webscraper.
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
anaskhan96/soup
Web Scraper in Go, similar to BeautifulSoup
benibela/xidel
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
scrapfly/scrapfly-scrapers
Scalable Python web scraping scripts for +40 popular domains
rootVIII/proxy_requests
a class that uses scraped proxies to make http GET/POST requests (Python requests)
salimk/Rcrawler
An R web crawler and scraper
onepointAI/onepoint
An AI assistant tool that integrates coding, writing, and reading functions. For better alternatives see https://monica.im/desktop
toby-p/rightmove_webscraper.py
Python class to scrape data from rightmove.co.uk and return listings in a pandas DataFrame object
intergalacticalvariable/reader
📚 This is an adapted version of Jina AI's Reader for local deployment using Docker. Convert any URL to an LLM-friendly input with a simple prefix http://127.0.0.1:3000/https://website-to-scrape.com/
serpapi/lego-ai-parser
Lego AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.
TBosak/mkfd
RSS feed builder created with Bun🥖 and Hono🔥- builds from webpages, email folders, and REST API calls.
AliAkhtari78/SpotifyScraper
Spotify Scraper to extract all the information from spotify, download mp3 with cover of the song
MichaelYochpaz/iSubRip
A Python command-line tool for scraping and downloading subtitles from AppleTV and iTunes movie pages.
mehmetozkaya/DotnetCrawler
DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c
bitsummation/pickaxe
SQL Based DSL Web Scraper/Screen Scraper
dwallach1/Stocker
Financial Web Scraper & Sentiment Classifier
chuanenlin/shutterscrape
Web scrapper for Shutterstock
CuriousLearner/GeeksForGeeksScrapper
Scrapes g4g and creates PDF
JesseVent/crypto
Cryptocurrency Historical Market Data R Package
hedii/php-crawler
A php crawler that finds emails on the internets
JonathanVusich/pcpartpicker
This is an unofficial API for the website pcpartpicker.com.
brandonrobertz/autoscrape-py
An automated, programming-free web scraper for interactive sites
nmcassa/letterboxdpy
A letterboxd webscraper
ayushi7rawat/CoWin-Vaccine-Notifier
Automated Python Script to retrieve vaccine slots availability and get notified when a slot is available.
EchterAlsFake/PHUB
A lightweight API for Pornhub
3xploitGuy/webscrape
A web scraper to scrape email's and phone numbers from Websites.
s-r-e-e-r-a-j/WebExtractor
WebExtractor is a powerful OSINT and ethical hacking tool developed in Python. It is used to extract email addresses, phone numbers, and links (both visible and hidden) from a target website
tech-engine/goscrapy
GoScrapy: Harnessing Go's power for blazingly fast web scraping, inspired by Python's Scrapy framework.
A-Wheeto/Dashboard
A tkinter GUI collating various data
mirkoschubert/gdpr-cli
A command line tool for checking your website for GDPR compliance.
RamonWill/price-comparison-project
A webscraper for the Django Framework that compares the product prices for various UK supermarkets
MakeYourLifeEasier/Wuxiaworld-2-eBook
This Python script will download chapters from novels availaible on wuxiaworld.com saves then into the .epub format
ZoranPandovski/BookingScraper
:earth_americas: :hotel: Scrape Booking.com :hotel: :earth_americas:
giuseppegambino/Scraping-TripAdvisor-with-Python-2020
Python implementation of web scraping of TripAdvisor with Selenium in a new 2019 website
iulspop/slack-web-scraper
Puppeteer configured to scrape the posts and threads of any channel on Slack.