/crawly

A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.

Primary LanguageRustMIT LicenseMIT

Watchers

No one’s watching this repository yet.