/spidy

A sqlite3 based web crawler

Primary LanguageCThe UnlicenseUnlicense

Spidy

A web spider for the sake of a web spider.

Features

  • follow html hrefs
  • explore / of any url
  • count references, no duplicates
  • remove php instructions