keylimetoolbox/resque-kubernetes

Use scaling for the jobs, rather than launching new jobs

Opened this issue · 0 comments

The current logic of this gem is to launch a new Kubernetes Job, with a single container (presuming the jobs has a single container defined in the manifest) for each time a resque job is enqueued. This leads to a lot of jobs.

We could use job scaling to handle this more cleanly, where we just increase the number of containers in the job as items are added to the queue, up to the maximum workers. See also, this walkthrough.

I'm not sure how we handle termination with this — because we'd want to decrease the number of containers needed as the containers wrap up when the queue empties out.