Question regarding reconciliation.
Closed this issue · 9 comments
Does reconciliation query for the status of a runner (i.e. active vs idle) or does it only check to see whether there is a registered runner for the underlying pod?
I ask because the operator doesn't seem to be scaling up when runners are in the active state. I would think that the automated scaling would check how many runners are busy and spawn new runners to ensure that there are enough idle runners to meet the spec requirement.
It checks the status. Check the controller logs for errors if it does not scale up/down.
The logs don't show any errors. I have my runners set up delete themselves after they complete a single job. The operator detects the deleted pods correctly and scales up to meet the minimum pool size of 2, but does not scale up when any of the runners are in an active state.
it depends a bit on the state of them in github - if they simply disappear they are probably still registered as alive, and maybe also still active.
Closing due to lack of activity.
Feel free to re-open, and add logs.
@davidkarlsen Just getting back to this now. I have confirmed that there is an issue with the operator differentiating between active vs idle runners.
I have my deployment configured to maintain a pool size of at least 3 and a max of 5. On startup, the operator correctly scales to 3 runners. If a runner disappears, then the operator will correctly respond and spawn a replacement runner.
However, the problem I'm seeing that when a runner enters an Active state, the operator does not detect this and so it does not scale up. I would expect the operator to detect that I have 1 active and 2 idle, and spawn an additional runner to maintain an available pool size of 3.
@blakjak44 It will scale up if the pool is saturated, the logic is here: https://github.com/evryfs/github-actions-runner-operator/blob/master/controllers/githubactionrunner_controller.go#L124
as long as you have idle runners and the total is equal to min it has no motivation to scale up.
GitHub
K8S operator for scheduling github actions runner pods - github-actions-runner-operator/githubactionrunner_controller.go at master · evryfs/github-actions-runner-operator
@davidkarlsen Ok are you interested in adding a configuration option that would allow setting the number of busy runners to be considered saturated? The reason I ask is because we are using your operator to manage a pool of VMs, and the startup time is significantly slower than for containers, so we would like to make sure we always have a pool of MinRunners available in an idle state.
@blakjak44 OK, I understand your usecase. So basically you're looking for a minIdle parameter in order to have that many free runners available? In contrast to https://github.com/evryfs/github-actions-runner-operator/blob/master/api/v1alpha1/githubactionrunner_types.go#L21
Can you create a new issue for this?
K8S operator for scheduling github actions runner pods - github-actions-runner-operator/githubactionrunner_types.go at master · evryfs/github-actions-runner-operator
@davidkarlsen Yes exactly. Opened issue #417. Thanks!