CaptainLazarus/spider

The fastest web crawler and indexer written in Rust.

Spider

The fastest web crawler and indexer.

Getting Started

View the spider or CLI directory to get started.

Benchmarks

See BENCHMARKS.

Examples

See EXAMPLES.

Contributing

See CONTRIBUTING.