stewartmckee/cobweb
Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
JavaScriptMIT
Issues
- 3
Feature request: Stop crawl at time
#54 opened by samnissen - 3
- 10
Falling into Crawl Traps
#53 opened by fuzzygroup - 1
undefined method `banner' for main:Object (NoMethodError) on calling from command line
#52 opened by gushonorato - 0
external_urls not treated as external
#16 opened by stewartmckee - 2
Error on first run
#48 opened by rap2hpoutre - 1
- 3
- 5
error while installing cobweb-1.0.28.gem: Invalid argument @ rb_sysopen
#41 opened by illtellyoulater - 1
How can I start stop crawling website
#46 opened by qiun - 6
Standalone Crawler gives error for redis
#25 opened by Shehbaz - 9
- 5
Encoding problems
#21 opened by wuiscmc - 3
- 2
LoadError with version 1.0.26
#39 opened by pisaacs - 3
- 0
Redirect Limit causing crawl to stop
#36 opened by stewartmckee - 1
Binary not installed
#35 opened by fabn - 1
Code organization
#34 opened by andrejj - 4
Thin web-server works slowly
#19 opened by sunloverz - 0
Inbound links are not normalized when stored
#29 opened by gh2k - 0
Improve connection handling
#24 opened by stewartmckee - 2
License missing from gemspec
#18 opened by bf4 - 4
Suggestion: Compatibility with Sidekiq
#17 opened by NebJ