[BUG] New version updates should roll out gracefully
Closed this issue · 3 comments
Describe the bug
Mixer is completely down and unresponsive when we roll out new versions. We should investigate doing rolling updates instead.
See also datacommonsorg/website#260
The mixer rolling out can be observed in https://pantheon.corp.google.com/kubernetes/deployment/us-central1-c/mixer-cluster/mixer/mixer-backend?organizationId=433637338589&project=datcom-mixer&tab=overview
From what I observed before, the 6 nodes will do rolling update. Need to investigate if any gke config is missing
Going to update the deployment.yaml based on the config here: https://medium.com/platformer-blog/enable-rolling-updates-in-kubernetes-with-zero-downtime-31d7ec388c81
Mixer rollout now uses readiness and liveness check and rollout strategy with 25% update rate: https://github.com/datacommonsorg/mixer/blob/master/deploy/base/deployment.yaml#L92-L100