Start Worker Dyno Closer to run_at

[Sammidysam]

It would be really cool if, instead of starting a worker right when a new job is created and stopping the worker when all jobs are completed, a worker is started a minute before a job, that job is completed, and then the worker is closed if there is not another job in the next minute. This would allow jobs that are to be run very far in the future, e.g. 5 days, to not cause massive payments by the user.

I don't know if this is actually possible, but it would be fantastic if it could be done.

[davidakachaos]

The problem is that there must some kind of timer be involved. How else can we start a worker?

[lyuzashi]

I had in mind to write a fork or patch to use an Event Machine timer in a thread which waits until (or close to) run_at before starting the worker process.
This idea may be beyond the scope of Workless or require introducing dependencies which may cause issues.
Once I have the time to write it I could submit a pull request if others are interested. I'd like to hear from @lostboy for any thoughts on this approach.

[paulcc]

Please see #106 - there is some middleware which regularly checks to see if scaling changes are needed, and we can use that mechanism to trigger wake-ups. Then you can change the any-jobs query to just look a few minutes into the future. We've used something like this for a few years!