A web crawler which visits every link on a given site and indexes them.
Primary LanguageGoMIT LicenseMIT