Archive Team

We Are Going To Rescue Your Shit

The Internet

Pinned Repositories

ArchiveBot
ArchiveBot, an IRC bot for archiving websites
Language:Python402 22 28479
grab-site
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Language:Python1.5k 39 210150
IA.BAK
We back up a lot of stuff from around the web; now it's time to back up the Internet Archive, just in case.
Language:Shell92 25 3522
parler-grab
Archiving Parler.
Language:Lua227 30 125
seesaw-kit
Making a reusable toolkit for writing seesaw scripts
Language:Python72 9 12334
Ubuntu-Warrior
Scripts to build and boot warrior virtual machine containing Docker
Language:Shell120 15 169
warrior-dockerfile
A Dockerfile for the ArchiveTeam Warrior
Language:Shell414 18 5365
warrior4-vm
Warrior virtual machine appliance (version 4)
Language:Rust54 8 121
wget-lua
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
Language:C129 20 1917
wpull
Wget-compatible web downloader and crawler.
Language:HTML595 22 46184

Archive Team's Repositories

ArchiveTeam/warrior-dockerfile
A Dockerfile for the ArchiveTeam Warrior
Language:Shell414 18 5365
ArchiveTeam/terroroftinytown
URLTeam's second generation of URL shortener archiving tools
Language:Python79 18 4715
ArchiveTeam/warrior4-vm
Warrior virtual machine appliance (version 4)
Language:Rust54 8 121
ArchiveTeam/ludios_wpull
wpull fork with fixes and faster parsing using html5-parser; used by grab-site; should go away when wpull is similarly improved
Language:HTML30 6 208
ArchiveTeam/youtube-grab
Archiving all metadata from YouTube (everything except videos themselves due to size)
Language:Lua28 9 132
ArchiveTeam/urls-grab
Archiving URLs (outlinks) from a variety of sources.
Language:Lua24 10 118
ArchiveTeam/telegram-grab
Archiving public telegram messages.
Language:Lua15 10 129
ArchiveTeam/urls-sources
Sources for urls-grab.
Language:Python10 6 1712
ArchiveTeam/blogger-grab
Archiving Blogger/Blogspot.
Language:Lua8 7 0
ArchiveTeam/megawarc
Nondestructive warc-in-tar to warc conversion
Language:Python8 13 64
ArchiveTeam/grab-base-df
Base Dockerfile for warrior project grab scripts
Language:Dockerfile5 15 54
ArchiveTeam/standalone-readme-template
Readme instructions template for manually running pipeline grab scripts outside the warrior
3 12 119
ArchiveTeam/fc2-items
Managing items for fc2-grab.
Language:Python1
ArchiveTeam/glitch-items
Managing items for glitch-grab.
Language:Python1
ArchiveTeam/goo-gl-items
Managing items for goo-gl-grab.
Language:Roff1 5 01
ArchiveTeam/sourceforgedeveloperweb-grab
Archiving SourceForge Developer Web.
Language:Lua1
ArchiveTeam/.github
ArchiveTeam/archiwumallegro-grab
Archiving Archiwum Allegro.
Language:Python
ArchiveTeam/archiwumallegro-items
Managing items for archiwumallegro-grab.
Language:Python
ArchiveTeam/askfm-items
Managing items for askfm-grab.
Language:Python
ArchiveTeam/gooblog-grab
Archiving gooブログ.
Language:Lua
ArchiveTeam/gooblog-items
Managing items for gooblog-grab.
Language:Python
ArchiveTeam/oshietegoo-grab
Archiving 教えて!goo.
Language:Lua2
ArchiveTeam/oshietegoo-items
Managing items for oshietegoo-grab.
ArchiveTeam/sourceforgedeveloperweb-items
Managing items for sourceforgedeveloperweb-grab.
Language:Python
ArchiveTeam/tistory-grab
Archiving Tistory.
Language:Lua
ArchiveTeam/tistory-items
Managing items for tistory-grab.
ArchiveTeam/twitch-items
Managing items for twitch-grab.
Language:Python6 0
ArchiveTeam/typepad-grab
Archiving Typepad.
Language:Lua
ArchiveTeam/typepad-items
Managing items for typepad-grab.
Language:Roff0 0