aichat-bot/crawly
A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.
RustMIT
Watchers
No one’s watching this repository yet.
A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.
RustMIT
No one’s watching this repository yet.