Queues scale temporarily for to -1 after autoscaler restart
aleclerc-sonrai opened this issue · 6 comments
I'm wondering if anyone has hit this, or seen this, or if it is expected behavior.
If the workerpodautoscaler pod restarts for any reason, upon startup, it temporarily sets the value on all of my wpas to -1, which then ends up scaling them to minReplicas (0 in my case). a few seconds later, it gets the actual queue size, updates the wpa and then scales up the deployment once more.
Is there some sort of race condition that I am hitting that might be causing this? I am using quite a few WPAs (over 100) so I'm partially thinking it might be a scale issue. I am planning on playing with some of the config variables today and digging through the code to see if I can replicate/debug what's happening.
It might be a bug. Thanks for reporting.
What do you mean by WPA to -1 here? Does it mean desired replica is being set to -1?
it temporarily sets the value on all of my wpas to -1,
Many values are initialised to -1 on startup. They are updated as we get their recent values. There could be a bug causing this, need to debug. Detailed logs of the event can help me troubleshoot.
I'll try to demo what I'm seeing with some more concrete examples.
But essentially, we run most of our workloads on AWS Spot instances, so then can go away/restart at any time (I've also seen a random error caused the pod to crash or restart).
I can easily see this if i restart the pod manually:
kubectl -n kube-system rollout restart deploy/workerpodautoscaler
So when that happens what I see:
k get wpa -ocustom-columns=NAME:.metadata.name,QUEUE:.status.CurrentMessages,DEPLOYMENTS:.status.AvailableReplicas,MIN:.spec.minReplicas,MAX:.spec.maxReplicas <redacted-wpa-name> -w
NAME QUEUE DEPLOYMENTS MIN MAX
<redacted-wpa-name> 3 1 0 4
<redacted-wpa-name> -1 1 0 4
<redacted-wpa-name> 2 0 0 4
<redacted-wpa-name> 0 1 0 4
and for the deployment
❯ k get deploy <redacted-deployment> -w
NAME READY UP-TO-DATE AVAILABLE AGE
<redacted-deployment> 1/1 1 1 7d
<redacted-deployment> 1/0 1 1 7d
<redacted-deployment> 1/0 1 1 7d
<redacted-deployment> 0/0 0 0 7d
<redacted-deployment> 0/1 0 0 7d
<redacted-deployment> 0/1 0 0 7d
<redacted-deployment> 0/1 0 0 7d
<redacted-deployment> 0/1 1 0 7d
<redacted-deployment> 1/1 1 1 7d
So essentially the queues gets set to -1, the deployment gets scaled to zero, all the pods terminate, it gets the queue value from sqs, and then scales the deployment back up to 1, new pods restart.
Also, as I mentioned, I have a LOT of wpa and deployments (in the hundreds) so this causes a huge swing of pod restrats across the entire cluster.
I would think, upon startup, that if a value currently exists, it wouldn't reset it back to -1 unless it was unsuccessfully able to get metrics from SQS, and then maybe then.
Thanks for the details.
-1 in queue size should not cause scale down. As it is uninitialized value.
Let me find the issue and fix the bug.
@aleclerc-sonrai Trying out a fix in https://github.com/practo/k8s-worker-pod-autoscaler/pull/137/files
Would you be able to use this image for your WPA pod and see if this fixes the issue?
practodev/workerpodautoscaler:v1.4.1-1-g73ba5c4
I just took that for a spin and it seems to solve the issue!
❯ ksys logs -f deploy/workerpodautoscaler | grep dev-inventa-root-user-queue
W0921 18:11:57.103341 1 controller.go:430] dev-queue qMsgs: -1 not uninitialized, waiting for init to complete
I0921 18:12:15.097447 1 controller.go:450] dev-queue qMsgs: 6869, desired: 4
I'll keep an eye out for when you cut a new version, but this is working great in the meantime.