Simple Crawler
A very simple crawler library
Quickstart
- Build container
docker build -t simple-crawler . - Run tests
docker run -v `pwd`:/usr/src/app simple-crawler cargo test --release
- Read the docs
cargo doc --open
Known issues
- This runs in memory for performance reasons. Seeing as it's built in rust memory usage is minimal but if using it on a site with 1000 or more urls stack overflow can occur if not run in release mode. Another option is to increase stack limit.
- A blocking option has been added which is enabled with the blocking feature. This will be removed over time as the async, concurrent and parallel options perform much better.