/spider

The fastest web crawler and indexer

Primary LanguageRustMIT LicenseMIT

Spider

Build Status Crates.io Documentation Rust

The fastest web crawler and indexer. Foundational building blocks for data curation workloads.

Getting Started

The simplest way to get started is to use the Spider Cloud for a pain free hosted service. View the spider or spider_cli directory for local installations.

Benchmarks

See BENCHMARKS.

Examples

See EXAMPLES.

License

This project is licensed under the MIT license.

Contributing

See CONTRIBUTING.