mikespook/gearman-go

Worker doesn't continue work after limit

jcorry opened this issue · 0 comments

worker.New() is documented:

//
// If limit is set to Unlimited(=0), the worker will grab all jobs
// and execute them parallelly.
// If limit is greater than zero, the number of paralled executing
// jobs are limited under the number. If limit is assgined to
// OneByOne(=1), there will be only one job executed in a time.

It's not working that way for me...

w := worker.New(10)
defer w.Close()

w.ErrorHandler = func(e error) {
	app.errorLog.Println(e)
	if opErr, ok := e.(*net.OpError); ok {
		if !opErr.Temporary() {
			proc, err := os.FindProcess(os.Getpid())
			if err != nil {
				app.errorLog.Println(err)
			}
			if err := proc.Signal(os.Interrupt); err != nil {
				app.errorLog.Println(err)
			}
		}
	}
}

// Add the GM server
w.AddServer("tcp4", config.GearmanServer)

w.AddFunc("Photos", app.TintBlue, worker.Unlimited)

if err := w.Ready(); err != nil {
	log.Fatal(err)
	return
}

go w.Work()

signal.Bind(os.Interrupt, func() uint { return signal.BreakExit })
signal.Wait()

If I add more than 10 things to my gearman queue, the first 10 are processed by the app.TintBlue func, but work halts there...the rest of the items in the queue are never processed.

If I use worker.Unlimited as the limit I run out of memory (working with 7-12mb images being retrieved from S3 and manipulated).

What am I missing? Why can't I specify the number of parallel processes to run my worker in...but trust that that amount of workers will do all of the work?

Oddly, if I use worker.OneByOne as the limit, it processes 2 items then halts.