practo/k8s-worker-pod-autoscaler

Non existent queues leads to high cpu in WPA

alok87 opened this issue · 0 comments

When the queue specified in WPA is not present, WPA CPU shoots up. Since it continuously tries to find the non existent queue. If there are many non existent queues then the CPU can increase upto 100% cpu limit leading to CPU throttling of the WPA pod and resulting in scaling becoming very slow.

E0302 12:03:44.409670       1 sqs.go:292] Unable to find queue "prod-dx-doctor_appointment_ingest_queue", AWS.SimpleQueueService.NonExistentQueue: The specified queue does not exist for this wsdl version.

Proposed Solutions

  • Retry with a crashloop if the queue does not exist.

  • Put external validation on non existent queue. Not adding validations in WPA because queue creation can happen by an external job. WPA validation should be done by the external service which creates WPA for you.