/crawler

Multi-threaded web crawler in ruby

Primary LanguageRubyMIT LicenseMIT

Multi-threaded web crawler in ruby

Setup:

  • Create config/config.yml, setup database section like any other rails database
  • Run rake db:migrate to create database tables

Usage:

$ ruby crawler.rb <modules>

Available (demo) modules:

  • imdb_serie
  • imdb_episode