Upgrading deis from 2.5.0 to 2.6.0 fails on Kubernetes 1.4
felipejfc opened this issue · 13 comments
Just followed the instructions from here
This is the log:
$ helmc install workflow-{$DESIRED_WORKFLOW_RELEASE}
---> Running `kubectl create -f` ...
namespace "deis" configured
secret "builder-ssh-private-keys" created
secret "builder-key-auth" created
secret "django-secret-key" created
secret "database-creds" created
secret "logger-redis-creds" configured
secret "objectstorage-keyfile" created
secret "registry-secret" created
secret "deis-router-dhparam" created
serviceaccount "deis-builder" created
serviceaccount "deis-controller" created
serviceaccount "deis-database" created
serviceaccount "deis-logger-fluentd" created
serviceaccount "deis-logger" created
serviceaccount "deis-monitor-telegraf" created
serviceaccount "deis-nsqd" created
serviceaccount "deis-router" created
serviceaccount "deis-workflow-manager" created
service "deis-builder" created
service "deis-controller" created
service "deis-database" created
service "deis-logger-redis" configured
service "deis-logger" created
service "deis-monitor-grafana" created
service "deis-monitor-influxapi" created
service "deis-monitor-influxui" created
service "deis-nsqd" created
proto: no encoder for TypeMeta unversioned.TypeMeta [GetProperties]
proto: tag has too few fields: "-"
proto: no coders for struct *reflect.rtype
proto: no encoder for sec int64 [GetProperties]
proto: no encoder for nsec int32 [GetProperties]
proto: no encoder for loc *time.Location [GetProperties]
proto: no encoder for Time time.Time [GetProperties]
proto: no coders for intstr.Type
proto: no encoder for Type intstr.Type [GetProperties]
The Service "deis-router" is invalid.
* metadata.resourceVersion: Invalid value: "": must be specified for an update
* spec.clusterIP: Invalid value: "": field is immutable
[ERROR] Failed to upload manifests: exit status 1
pinging @tmc as I believe he has seen this specific issue before and knows a workaround :)
@felipejfc have you resolved this issue yourself? If so how did you resolve it? IIRC it has something to do with when you modify the deis-router service's annotations... Or something of that nature.
@bacongobbler I've created and replaced the deployments manually and didn't touch the router service as it had no upgrades
Can you please show your service manifest from the old release? That way we can try to identify which field cannot be updated and hopefully we can come up with a fix.
@bacongobbler Here it is:
$ kubectl edit svc deis-router --namespace deis
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
annotations:
chart.helm.sh/description: Deis Workflow
chart.helm.sh/file: /home/felipe/.helmc/workspace/charts/workflow-v2.5.0/manifests/deis-router-service.yaml
chart.helm.sh/name: workflow-v2.5.0
chart.helm.sh/version: v2.5.0
helm-keep: "true"
kubectl.kubernetes.io/last-applied-configuration: '{"kind":"Service","apiVersion":"v1","metadata":{"name":"deis-router","namespace":"deis","selfLink":"/api/v1/namespaces/deis/services/deis-router","uid":"3cbf8d61-5b52-11e6-8414-0a656e9cbc8d","resourceVersion":"6906941","creationTimestamp":"2016-08-05T21:19:04Z","labels":{"heritage":"deis"},"annotations":{"chart.helm.sh/description":"Deis
Workflow","chart.helm.sh/file":"/home/felipe/.helmc/workspace/charts/workflow-v2.5.0/manifests/deis-router-service.yaml","chart.helm.sh/name":"workflow-v2.5.0","chart.helm.sh/version":"v2.5.0","helm-keep":"true"}},"spec":{"ports":[{"name":"http","protocol":"TCP","port":80,"targetPort":8080,"nodePort":30372},{"name":"https","protocol":"TCP","port":443,"targetPort":6443,"nodePort":32228},{"name":"builder","protocol":"TCP","port":2222,"targetPort":2222,"nodePort":30928},{"name":"healthz","protocol":"TCP","port":9090,"targetPort":9090,"nodePort":31437}],"selector":{"app":"deis-router"},"clusterIP":"100.71.77.67","type":"LoadBalancer","sessionAffinity":"None"},"status":{"loadBalancer":{"ingress":[{"hostname":"xxxxxxxxxxxxxxxxxxxxx.us-east-1.elb.amazonaws.com"}]}}}'
creationTimestamp: 2016-08-05T21:19:04Z
labels:
heritage: deis
name: deis-router
namespace: deis
resourceVersion: "6909803"
selfLink: /api/v1/namespaces/deis/services/deis-router
uid: 3cbf8d61-5b52-11e6-8414-0a656e9cbc8d
spec:
clusterIP: 100.71.77.67
ports:
- name: http
nodePort: 30372
port: 80
protocol: TCP
targetPort: 8080
- name: https
nodePort: 32228
port: 443
protocol: TCP
targetPort: 6443
- name: builder
nodePort: 30928
port: 2222
protocol: TCP
targetPort: 2222
- name: healthz
nodePort: 31437
port: 9090
protocol: TCP
targetPort: 9090
selector:
app: deis-router
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- hostname: xxxxxxxxxxxxxxxxxx.us-east-1.elb.amazonaws.com
I ran into the same issue while upgrading from deis v2.6 to v2.7 on k8s v1.2.6. I deleted the deis-router service
kubectl delete svc deis-router --namespace deis
and succesfully ran the installation again. Afterwards I edited the deis-router service and set the nodePorts back to the values matching my current LB setup.
I'm seeing health check failures on some of the deis router pods in 1 my environments. These are now running 2.6.4 the issue started appearing for me some where after 2.6.0 I believe. This is on K8S 1.4.0
deis-router-1301926735-2cnfe 1/1 Running 14 2d
deis-router-1301926735-g2d5y 1/1 Running 1 2d
deis-router-1301926735-goieo 1/1 Running 2 2d
deis-router-1301926735-xexd1 1/1 Running 56 2d
@sstarcher can you please post a new issue? That doesn't seem related to the OP's issue.
@bacongobbler I upgraded to 2.6.5 and the problem seems to have ceased. I'm not sure what the root cause was for the issue.
Perhaps 646b1c6?
EDIT: sorry, deis/charts#368
@bacongobbler My issue seemed to be with the healthz failing after a while. The initial delay would not affect a currently running container would it?
Not a long-running container, no.
Closing as fixed with helm v2.