Conceptual Question: Do I need to spin up external workers?
tonyarnold opened this issue · 11 comments
Sorry if this is a bit of an obtuse question, but I'm a bit stumped - do I need to run another vapor instance alongside my "main" instance to run jobs?
If they do need to be spun up externally, how are you deploying instances like this? A completely separate application instance pointing to your original database?
No apologies necessary! Yes typically you would run this alongside your main application. If you're using something like Heroku, the concept of Dynos applies here - you would have the application dynos running your main code and then some worker dynos (an arbitrary amount, specified by you) that run the queue worker. They both run off of the same codebase but the initial start command differs.
With the Vapor 4 version we are also going to support in-process running so that you can run the jobs worker in the same process as the main application, removing the need for parallel deployments. We have to write the documentation still but I believe you'll either be able to pass a flag to the JobsProvider or you can manually start it using JobsWorker(...).start()
Hope this helps! Feel free to reopen if anything else comes up.
With the Vapor 4 version we are also going to support in-process running so that you can run the jobs worker in the same process as the main application […]
I haven't looked at v4 yet - is there a technical reason why this feature couldn't be back ported to v3?
I don't think there's a technical reason but it's definitely non-trivial. This is what the file looks like currently: https://github.com/vapor/jobs/blob/3/Sources/Jobs/JobsCommand.swift and the new version splits it into a JobsWorker.
Awesome, thanks for the info - I might have a crack at back porting it in a week or two.
Cool, I think most would find that a welcome addition 😊👍🏻
We have to write the documentation still but I believe you'll either be able to pass a flag to the
JobsProvideror you can manually start it usingJobsWorker(...).start()
Did this get worked out, or is it in the code right now? Is there an example I can have a look at?
On this class: https://github.com/vapor/jobs/blob/master/Sources/Jobs/JobsWorker.swift#L6 you should be able to call .start(on: queue)
(Although I think ideally we'd add a flag to the provider and handle it in the background)
So the main app need to retain the worker instances somewhere/somehow? In boot, maybe?
Do you have a design in mind for how the app and the jobs queue would co-exist in the same process?
I'm guessing something like this? https://gist.github.com/tonyarnold/c62b8ef70e0b3fe5f01cfa3e97d18233
Yeah that looks nice! This line: https://gist.github.com/tonyarnold/c62b8ef70e0b3fe5f01cfa3e97d18233#file-inprocessjobsprovider-swift-L28 we should make customizable. But overall that's exactly what I had in mind