PuerkitoBio/fetchbot
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
GoBSD-3-Clause
Issues
- 0
Update package to use go.mod
#33 opened by AdamSLevy - 0
- 1
robotstxt-go has renamed to robotstxt
#31 opened by michael-stevens - 1
HeaderProvider example
#30 opened by ernsheong - 5
Parallellize queue
#29 opened by arthurgustin - 3
Add a random delay between each cmd?
#28 opened by AllenDang - 3
- 1
Cancel() make goroutine leak
#26 opened by ryu-koui - 2
Getting lots of i/o timeouts
#23 opened - 3
Expose queue size
#21 opened by FnuGk - 3
- 4
Fail queue object in handler
#18 opened by clanstyles - 1
- 8
Drain queue
#10 opened by grafana-dee - 1
Handler and Matcher Design
#11 opened by mmcdole - 1
q.Block even if seed empty
#5 opened by pilere - 3
limit the depth
#1 opened by fils