All Masters Never Become Ready
dgoodwin opened this issue · 8 comments
dgoodwin commented
Multiple folks on our team are seeing this today.
Using an image built from: openshift/installer@08018ca
We end up completing install, mostly:
level=debug msg="API not up yet: the server could not find the requested resource"
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 54.82.254.148:6443: connect: connection refused"
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 52.1.146.45:6443: connect: connection refused"
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 54.82.254.148:6443: connect: connection refused"
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 52.3.185.104:6443: connect: connection refused"
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 54.159.140.198:6443: connect: connection refused"
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 34.197.191.119:6443: connect: connection refused"
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 54.158.200.103:6443: connect: connection refused"
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 52.1.146.45:6443: connect: connection refused"
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 54.82.254.148:6443: connect: connection refused"
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 52.3.185.104:6443: connect: connection refused"
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 54.159.140.198:6443: connect: connection refused"
level=info msg="API v1.11.0+d4cacc0 up"
level=debug msg="added kube-scheduler.1566c06fe7813814: ip-10-0-0-76_c653648b-e76c-11e8-8d31-0ec82e7bcc70 became leader"
level=debug msg="added kube-controller-manager.1566c070f2c6967b: ip-10-0-0-76_c6645265-e76c-11e8-860d-0ec82e7bcc70 became leader"
level=warning msg="RetryWatcher - getting event failed! Re-creating the watcher. Last RV: 219"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 52.1.146.45:6443: connect: connection refused"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 54.159.140.198:6443: connect: connection refused"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 52.1.146.45:6443: connect: connection refused"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 52.3.185.104:6443: connect: connection refused"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 34.197.191.119:6443: connect: connection refused"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 54.158.200.103:6443: connect: connection refused"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 54.82.254.148:6443: connect: connection refused"
level=error msg="waiting for bootstrap-complete: watch closed before UntilWithoutRetry timeout"
level=info msg="Install complete! Run 'export KUBECONFIG=/output/auth/kubeconfig' to manage your cluster."
level=info msg="After exporting your kubeconfig, run 'oc -h' for a list of OpenShift client commands."
Masters are permanently stuck notready:
$ k get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-19-84.ec2.internal NotReady master 1h v1.11.0+d4cacc0
ip-10-0-38-133.ec2.internal NotReady master 1h v1.11.0+d4cacc0
ip-10-0-8-183.ec2.internal NotReady master 1h v1.11.0+d4cacc0
Their status contains a condition like:
- lastHeartbeatTime: 2018-11-13T18:59:05Z
lastTransitionTime: 2018-11-13T17:52:16Z
message: 'runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady
message:Network plugin returns error: cni config uninitialized'
reason: KubeletNotReady
status: "False"
type: Ready
With a similar error in the kubelet systemd logs:
Nov 13 19:00:12 ip-10-0-8-183 hyperkube[841]: E1113 19:00:12.951372 841 kubelet.go:2101] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config uninitialized
cluster-network-operator pod logs show:
E1113 18:53:06.044313 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get https://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.12.190:6443: connect: connection refused
2018/11/13 18:53:14 Reconciling NetworkConfig /default
squeed commented
Odd - it got a network configuration, but just... hung. It uses the stock operator-sdk. @shawn-hurley, any thoughts how this could happen?
What does kubectl get -n openshift-sdn pods show? Did the operator make any progress at all?
dgoodwin commented
No openshift-sdn namespace in this cluster. Is that bad?
squeed commented
The next time this happens, watch the cluster-network-operator to see if its making progress. If not, we need to find out why.
dgoodwin commented
Still happening this morning:
$ klf cluster-network-operator-z8bc6 [12/3643]
2018/11/14 11:38:36 Go Version: go1.10.3
2018/11/14 11:38:36 Go OS/Arch: linux/amd64
2018/11/14 11:38:36 operator-sdk Version: v0.1.0
2018/11/14 11:38:36 overriding kubernetes api to https://dgoodwin1-api.new-installer.openshift.com:6443
time="2018-11-14T11:38:36Z" level=info msg="trying to become the leader"
time="2018-11-14T11:38:38Z" level=info msg="No pre-existing lock was found."
time="2018-11-14T11:38:38Z" level=info msg="Became the leader."
2018/11/14 11:38:39 Registering Components.
2018/11/14 11:38:39 Starting the Cmd.
2018/11/14 11:38:39 Reconciling NetworkConfig /default
2018/11/14 11:38:40 Reconciling NetworkConfig /default
2018/11/14 11:38:41 Reconciling NetworkConfig /default
2018/11/14 11:38:42 Reconciling NetworkConfig /default
2018/11/14 11:38:43 Reconciling NetworkConfig /default
2018/11/14 11:38:44 Reconciling NetworkConfig /default
2018/11/14 11:38:45 Reconciling NetworkConfig /default
2018/11/14 11:38:46 Reconciling NetworkConfig /default
2018/11/14 11:38:47 Reconciling NetworkConfig /default
2018/11/14 11:38:48 Reconciling NetworkConfig /default
2018/11/14 11:38:51 Reconciling NetworkConfig /default
2018/11/14 11:38:56 Reconciling NetworkConfig /default
2018/11/14 11:39:06 Reconciling NetworkConfig /default
2018/11/14 11:39:27 Reconciling NetworkConfig /default
2018/11/14 11:40:08 Reconciling NetworkConfig /default
2018/11/14 11:41:30 Reconciling NetworkConfig /default
2018/11/14 11:44:14 Reconciling NetworkConfig /default
2018/11/14 11:49:41 Reconciling NetworkConfig /default
E1114 11:57:59.896544 1 reflector.go:322] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to watch *v1.NetworkConfig: Get
https://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?resourceVersion=255&timeoutSeconds=506&watch=true: dial tcp 10.0.1.209:6443: connect: connection refused
E1114 11:58:00.979228 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.39.68:6443: connect: connection refused
E1114 11:58:01.991688 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.1.209:6443: connect: connection refused
E1114 11:58:03.004932 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.39.68:6443: connect: connection refused
E1114 11:58:04.019185 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.93.45:6443: connect: connection refused
E1114 11:58:05.079807 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.27.254:6443: connect: connection refused
E1114 11:58:06.178972 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.60.129:6443: connect: connection refused
E1114 11:58:07.191016 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.93.45:6443: connect: connection refused
E1114 11:58:08.203615 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.27.254:6443: connect: connection refused
E1114 11:58:09.216142 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.60.129:6443: connect: connection refused
E1114 11:58:10.229218 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.93.45:6443: connect: connection refused
E1114 11:58:11.242174 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.27.254:6443: connect: connection refused
E1114 11:58:12.254245 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.60.129:6443: connect: connection refused
E1114 11:58:14.281861 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.79.187:6443: connect: connection refused
E1114 11:58:15.294857 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.1.209:6443: connect: connection refused
E1114 11:58:23.908344 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: netwo
rkconfigs.networkoperator.openshift.io is forbidden: User "system:serviceaccount:openshift-cluster-network-operator:default" cannot list networkconfigs.networkoperator.openshift.io at the cluster scope: no RBAC p
olicy matched
2018/11/14 11:58:24 Reconciling NetworkConfig /default
2018/11/14 12:00:37 Reconciling NetworkConfig /default
E1114 12:18:29.475698 1 reflector.go:322] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to watch *v1.NetworkConfig: Get
https://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?resourceVersion=2949&timeoutSeconds=383&watch=true: dial tcp 10.0.60.129:6443: connect: connection refuse
d
E1114 12:18:30.488786 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.93.45:6443: connect: connection refused
E1114 12:18:31.501424 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.27.254:6443: connect: connection refused
E1114 12:18:32.519018 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.60.129:6443: connect: connection refused
E1114 12:18:33.531669 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.93.45:6443: connect: connection refused
E1114 12:18:34.568049 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.27.254:6443: connect: connection refused
E1114 12:18:35.580845 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.79.187:6443: connect: connection refused
E1114 12:18:36.593299 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.1.209:6443: connect: connection refused
E1114 12:18:37.677087 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.39.68:6443: connect: connection refused
E1114 12:18:38.688812 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.79.187:6443: connect: connection refused
E1114 12:18:39.703323 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.1.209:6443: connect: connection refused
E1114 12:18:40.715925 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.39.68:6443: connect: connection refused
E1114 12:18:41.766929 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.79.187:6443: connect: connection refused
E1114 12:18:42.777297 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: the s
erver could not find the requested resource (get networkconfigs.networkoperator.openshift.io)
E1114 12:18:43.790260 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.39.68:6443: connect: connection refused
E1114 12:18:44.803018 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.60.129:6443: connect: connection refused
E1114 12:18:53.504680 1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: the s
erver could not find the requested resource (get networkconfigs.networkoperator.openshift.io)
dgoodwin commented
May be an issue on our side, we're running openshift-install in pod containers, and we're seeing failure there but success running the container manually with the exact same image. Please hold on any major investigation.
squeed commented
Yeah, the cluster-network-operator is a very special snowflake. It is run before the bootstrap apiserver is torn down. That means it's running on the masters, talking to the boostrap control plane.
In some sense, it's the canary in the coal mine.
squeed commented
I've not seen any major issues with the openshift-sdn for a while; closing this. If you see more issues please open another bug.
dgoodwin commented
Yeah this was on our end. Thanks!