vertical-pod-autoscaler: updater suddenly evicted a lot of pods after being redeployed
milesbxf opened this issue · 4 comments
Raising an issue separately to #2178 to track what we've found
We had an issue with the VPA recently where we redeployed the recommender and updater; the updater immediately started evicting pods from nearly half our cluster (2-3k pods) in about 15-20 minutes. We're discussing how to mitigate the effects of pod/endpoint churn in #2178; this issue is about why the mass evictions happened.
What we did in the build up to the issue
We have the vertical pod autoscaler components set up to autoscale themselves (:inception:). The recommender had recommended over 1GiB memory, but the admission controller was unable to override the pod requests, as we still had pod resource limits set (to 1GiB).
We removed the limits and redeployed both components. The admission controller successfully updated both pods with new requests.
Recommender:
CPU 63m -> 203m
Memory 1000MiB -> 1GiB
Updater:
CPU 49m -> 126m
Memory 878Mi -> 1GiB
What we expected to happen
No change in pod eviction rate when redeploying the recommender/updater
What actually happened
Lots of pods were evicted:
Other interesting info
The recommender execution latency dropped by half after the pod was redeployed with the new resource requests:
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen.
Mark the issue as fresh with/remove-lifecycle rotten.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.