Add Mercator URL Frontier implementation

Question

Add Mercator URL Frontier implementation

moredure opened this issue 5 years ago · 5 comments

@BruceDone checkout #62 please, added reference to https://dev.to/spacenomad/mercator-url-frontier-in-golang-15o2

Answer 1 · 2020-11-06T03:43:32.000Z

Is this a framework ? @moredure

Answer 2 · 2020-11-08T14:56:21.000Z

Hello @BruceDone, I have some ideas to wrap it into a framework, but generally it is just a reference implementation of Mercator Crawler URL Frontier in Golang, which allows to conduct smart (calculate windows between requests to the same host based on previous request duration to the same host or some static values, etc) and polite crawling as well as to enable crawling without memory limitations for URL queues both seed and collected during crawling.

Answer 3 · 2020-11-08T14:59:00.000Z

Should it be a complete framework to pass your moderation @BruceDone , or just a useful reference will be enought?

Answer 4 · 2020-11-09T07:00:17.000Z

Thanks for your contribute, but this repo just collect the crawler framework

Answer 5 · 2020-11-09T21:07:18.000Z

Understood, thanks for your reply.