/labyrinth

Crawl it.. I dare you - I double dare you!

Primary LanguageJavaScript

Labyrinth

Have you ever crawled a site? You probably have.

Now try to crawl this site. It should take some time :)

Installing

  • Use rvm to use Ruby 2.1.0
  • Run gem install bundler
  • Run bundler Possibly from the rvm install dir
  • Run rake:setup
  • Run rake:migrate
  • Run rackup

TODO

  • Add logging for requested urls
  • Use something not SQLite for logging This is real!
  • Cache the wordlist to give the disc some peace