/crawler-go

Primary LanguageGo

Crawler

A site crawler written in GO.

TODO

  1. Refactor to OO under test
  2. Normalize all links and assets (turn into absolute urls)
  3. Filter external links out
  4. JSON output
  5. Recurse or queue to scrape all links
  6. Concurrency
  7. Intermediate storage (resuming)