johnnagro/spider

Issues

• Follow vs nofollow links

  #11 opened a year ago by strzibny
  0

• Inefficient URL scanning

  #10 opened 5 years ago by apfeltee
  0

• Parse URL twice when found anchor

  #8 opened 5 years ago by madeindjs
  0

• Is there a way to add own URLs down the road while crawling is taking place?

  #4 opened 6 years ago by arslanaly47
  1

• /lib/spider/robot_rules.rb is undistributable

  #1 opened 8 years ago by eribertomota
  3