/fetchbot

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

Primary LanguageGoBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

Watchers