web-parser

Prepare

  • bundle install
  • Initialize URLs in application.yml
  • mkdir tmp for cache

run

  • with params timeout, max_retry, pool_size -- ruby index.rb 10 5 2
  • without params -- ruby index.rb