Implement a task queue with fixed number of workers.

Question

Implement a task queue with fixed number of workers.

Closed this issue 6 years ago · 2 comments

Currently, an new goroutine is launched for every collected/filtered URL. The number of goroutines are hence uncontrolled. Implement a task queue with fixed number of workers that can be specified by the user through commandline.

Answer 1 · 2018-06-22T18:06:44.000Z

https://github.com/Jeffail/tunny

Answer 2 · 2018-06-23T15:33:02.000Z

Looks useful. I'll try it in a separate branch.

Also, I have implemented a solution in the executor package. It's simple and highly decoupled from the crawler logic.

Take a look https://github.com/farkaskid/WebCrawler/tree/add-task-queue?files=1