Pinned Repositories
urlcanon
url canonicalization library for python and java
brozzler
brozzler - distributed browser-based web crawler
doublethink
rethinkdb python library
heritrix3
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
warcprox
WARC writing MITM HTTP/S proxy
warctools
Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)
brozzler
brozzler - distributed browser-based web crawler
libhdfs3-deb
Steps to produce a .deb that can be installed dpkg for libhdfs3 (for use with python)
monie
man-in-the-middle http/https proxy library in rust
warc
Read and write WARC files in Go
nlevitt's Repositories
nlevitt/monie
man-in-the-middle http/https proxy library in rust
nlevitt/libhdfs3-deb
Steps to produce a .deb that can be installed dpkg for libhdfs3 (for use with python)
nlevitt/warc
Read and write WARC files in Go
nlevitt/brozzler
brozzler - distributed browser-based web crawler
nlevitt/umbra
A queue-controlled browser automation tool for improving web crawl quality
nlevitt/warcio-rs
nlevitt/warcprox-rs
nlevitt/calstrsdivest
nlevitt/cdxj-indexer
CDXJ Indexing of WARC/ARCs
nlevitt/hdfs
A native go client for HDFS
nlevitt/heritrix3
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
nlevitt/hyper
An HTTP library for Rust
nlevitt/logaggrd
simple udp log aggregator daemon
nlevitt/lru-cache
A cache that holds a limited number of key-value pairs
nlevitt/outbackcdx
A Wayback RemoteResourceIndex server using RocksDB
nlevitt/pretty-env-logger
A pretty, easy-to-use logger for Rust.
nlevitt/psutil
A cross-platform process and system utilities module for Python
nlevitt/psutilz
utilities built on the psutil library
nlevitt/pywb
nlevitt/rocksdb
A library that provides an embeddable, persistent key-value store for fast storage.
nlevitt/snakebite
A pure python HDFS client
nlevitt/sshrc
bring your .bashrc, .vimrc, etc. with you when you ssh
nlevitt/swift-nio
Event-driven network application framework for high performance protocol servers & clients, non-blocking.
nlevitt/swift-nio-examples
examples of how to use swift-nio
nlevitt/tempfile
Temporary file library for rust
nlevitt/tokio
A runtime for writing reliable, asynchronous, and slim applications with the Rust programming language.
nlevitt/trough
Trough: Big data, small databases.
nlevitt/urlcanon
url canonicalization library for python and java
nlevitt/warcio
Streaming WARC/ARC library for fast web archive IO
nlevitt/warcprox
WARC writing MITM HTTP/S proxy