Spawning worker(s) at app start
mwf opened this issue · 4 comments
Hi!
IMHO, it would be usefull to have some option in app's Profile to autospawn configured amount of workers at app start. And keep them alive despite of idle-timeout option. All extra-workers should be killed after timeout.
Spawning new workers is not a quick process (est. 200-300 ms), and it can be unpleasant in some cases.
And the second issue.
we can set idle-timeout to zero - spawned workers will never die. If you don't like my first idea - maybe, implement it only for idle-timeout = 0 case? If the workers never die, it's no matter when to start them - at app start or at first query.
What do you think about it?
Hi.
Spawning new worker may be an issue only on cold-start of app, because if you already have some workload it means that you already have enough workers to handle that workload. Cold-start is rare situation so we don't really think that's an issue. And on the other hand, having some amount of idle workers is unnecessary spending of resources (i.e memory and connections). Also, we don't know and can't know how to calculate the right number of pre-spawned app workers without real workload, and it can not be configured via profile, because that number should depend on workload. Maybe sometimes in the future it will be done with some heuristic, but not with simple worker pre-spawning.
About second issue. What's the point of having app workers being idle?
Spawning new worker may be an issue only on cold-start of app, because if you already have some workload it means that you already have enough workers to handle that workload. Cold-start is rare situation so we don't really think that's an issue.
Cold-start is rare only if you HAVE some "constant" workload =) And if it's rather instable, first-worker starts will occur from time to time... For someone they may be a little annoying.
This
And on the other hand, having some amount of idle workers is unnecessary spending of resources (i.e memory and connections).
and this
Also, we don't know and can't know how to calculate the right number of pre-spawned app workers without real workload, and it can not be configured via profile, because that number should depend on workload.
is admin's business, I think... If he really needs more smooth app-start - he will be able to have it with this feature =) You give the opportunity - admin decides whether to use it, you see?
About second issue. What's the point of having app workers being idle?
I don't know what's the point to have workers idle forever. But it was your team who made this feature =)
It's just the same thoughts about opportunities. You've given a way to start non-dying workers through this config param - then it's rather obvious to give a way of pre-spawning this forever-living workers.
This all is not a kind of some important feature, it's just my thoughts I've decided to share with you. I've decided, maybe they can be useful somehow =)
Oh, I missed that somehow.
But it was your team who made this feature
Frankly, it looks more like a bug than a feature : ) I guess, we'll fix it in the nearest future.
Yes, @noxiouz ?
Hi.
Immortal workers feature was added for some reason, but nobody remembers what reason it was, so we'll probably drop it, yeah. We are planning to rework the scheduler, and the current implementation ideas for that new scheduler do not go well with pre-spawned or immortal workers, mainly because we plan to migrate workers around the cluster to utilize resources more effectively, which kind of kills the 'always ready' part of the benefits of having a pre-spawned worker pool, and I don't see any other benefits anyway.