Worker doesn't continue work after limit
jcorry opened this issue · 0 comments
jcorry commented
worker.New() is documented:
//
// If limit is set to Unlimited(=0), the worker will grab all jobs
// and execute them parallelly.
// If limit is greater than zero, the number of paralled executing
// jobs are limited under the number. If limit is assgined to
// OneByOne(=1), there will be only one job executed in a time.
It's not working that way for me...
w := worker.New(10)
defer w.Close()
w.ErrorHandler = func(e error) {
app.errorLog.Println(e)
if opErr, ok := e.(*net.OpError); ok {
if !opErr.Temporary() {
proc, err := os.FindProcess(os.Getpid())
if err != nil {
app.errorLog.Println(err)
}
if err := proc.Signal(os.Interrupt); err != nil {
app.errorLog.Println(err)
}
}
}
}
// Add the GM server
w.AddServer("tcp4", config.GearmanServer)
w.AddFunc("Photos", app.TintBlue, worker.Unlimited)
if err := w.Ready(); err != nil {
log.Fatal(err)
return
}
go w.Work()
signal.Bind(os.Interrupt, func() uint { return signal.BreakExit })
signal.Wait()
If I add more than 10 things to my gearman queue, the first 10 are processed by the app.TintBlue
func, but work halts there...the rest of the items in the queue are never processed.
If I use worker.Unlimited
as the limit I run out of memory (working with 7-12mb images being retrieved from S3 and manipulated).
What am I missing? Why can't I specify the number of parallel processes to run my worker in...but trust that that amount of workers will do all of the work?
Oddly, if I use worker.OneByOne
as the limit, it processes 2 items then halts.