practo/k8s-worker-pod-autoscaler

Deployment gets stuck MinimumReplicasUnavailable

mihaivinaga opened this issue · 2 comments

mihaivinaga commented

I have upgraded to the last library and since then I am having issues with the autoscaler. What happens is this after a period of time the pod that does the scaling gets stuck in pending. This is the error that I am getting:

Get "https://10.70.152.3:10250/containerLogs/kube-system/workerpodautoscaler-f898df7cb-wdsb7/wpa?tailLines=5000×tamps=true": dial tcp 10.70.152.3:10250: connect: no route to host

When I check the autoscaler deployment I get another message:

MinimumReplicasUnavailable
Deployment does not have minimum availability

This started happening after I've upgraded to the last version. It cannot be a problem of availability because I have 50+ nodes in kubernetes cluster. The only way that I get it to work is if I delete the cluster and recreate it from scratch.

Any pointers on what might be causing this?

alok87 commented

First search result on no route to host.

This error indicates that the Kubernetes API server is unable to reach the node. This has nothing to do with your containers, pods or even your CNI network. no route to host indicates that either:

The host is unavailable
A network segmentation has occurred
The Kubelet is unable to answer the API server

Looks like an k8s environment issue..

These environment related information can help me help you:

  • Kubernetes environment, version you are using?
  • WPA version? are you are v1.1.0? Did you follow this?
  • Output of kubectl get deployment workerpodautoscaler -o yaml -n kube-system
  • Is your deployment.yaml same as this ?
  • Is your CRD same as this ?
  • Can you share the Kubernetes Deployment resource and WPA resource (if possible)?
kubectl get deploy <ur_deploy_name> -n <namespace> -o yaml
kubectl get wpa <ur_wpa_name> -n <namespace> -o yaml

Also,

Try these latest released images today (tags like v1 and v1.1 always gets overwritten with the latest, recommended to use the exact tag like v1.1.0-18-gd098b97)

pushed: practodev/workerpodautoscaler:v1.1.0-18-gd098b97
pushed: practodev/workerpodautoscaler:v1.1
pushed: practodev/workerpodautoscaler:v1
alok87 commented

Closing this due to inactivity and does not look like a WPA bug.