warc
There are 110 repositories under warc topic.
ArchiveBox/ArchiveBox
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
internetarchive/heritrix3
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
ArchiveTeam/grab-site
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Rhizome-Conifer/conifer
Collect and revisit web pages.
webrecorder/archiveweb.page
A High-Fidelity Web Archiving Extension for Chrome and Chromium based browsers!
webrecorder/browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
webrecorder/replayweb.page
Serverless replay of web archives directly in the browser
oduwsdl/ipwb
InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS
webrecorder/webrecorder-player
Webrecorder Player for Desktop (OSX/Windows/Linux). (Built with Electron + Webrecorder)
webrecorder/warcio
Streaming WARC/ARC library for fast web archive IO
Florents-Tselai/WarcDB
WarcDB: Web crawl data as SQLite databases.
machawk1/wail
:whale2: Web Archiving Integration Layer: One-Click User Instigated Preservation
commoncrawl/news-crawl
News crawling with StormCrawler - stores content as WARC
webrecorder/browsertrix
Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!
bitextor/bitextor
Bitextor generates translation memories from multilingual websites
harvard-lil/warc-gpt
WARC + AI - Experimental Retrieval Augmented Generation Pipeline for Web Archive Collections.
machawk1/warcreate
Chrome extension to "Create WARC files from any webpage"
cocrawler/cocrawler
CoCrawler is a versatile web crawler built using modern tools and concurrency.
cocrawler/cdx_toolkit
A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
helgeho/ArchiveSpark
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
crissyfield/troll-a
Drill into WARC web archives
ArchiveTeam/wget-lua
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
N0taN3rd/wail
:whale2: One-Click User Instigated Preservation
chatnoir-eu/chatnoir-resiliparse
A robust web archive analytics toolkit
maxcountryman/warc-parquet
🗄️ A simple CLI for converting WARC to Parquet.
N0taN3rd/node-warc
Parse And Create Web ARChive (WARC) files with node.js
mikwielgus/forum-dl
Scrape posts, threads from forums, news aggregators, mail archives, export to JSONL, mailbox, WARC
CGamesPlay/chronicler
Offline-first web browser
internetarchive/cdx-summary
Summarize web archive capture index (CDX) files.
centic9/CommonCrawlDocumentDownload
A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika
openzim/warc2zim
Command line tool to convert a file in the WARC format to a file in the ZIM format
pirate/internet-archiving-talk
🎭 An introduction to the Internet Archiving ecosystem, tooling, and some of the ethical dilemmas that the community faces.
jedireza/warc
:gear: A Rust library for reading and writing WARC files
archivesunleashed/warclight
A Rails engine supporting the discovery of web archives.
PromyLOPh/crocoite
Web archiving using Google Chrome
nlnwa/warchaeology
Command line tool for digging into WARC files