Error running command - could not find a ready tiller pod
yoplait opened this issue · 1 comments
yoplait commented
Hi team, looks like there are some issues with the scripts:
null_resource.kube-cni:` Provisioning with 'local-exec'...
null_resource.kube-cni (local-exec): Executing: ["/bin/sh" "-c" "KUBECONFIG=secrets/admin.conf helm install -n kube-system hcloud-csi-driver mlohr/hcloud-csi-driver --set csiDriver.secret.create=true --set csiDriver.secret.hcloudApiToken=0lQ5BEtHPUxodken3TCqF6pR7ZA112DSFmf5K71mEqM9YVUOSIiOj8Kt68LNM2bV"]
null_resource.kube-cni (local-exec): Error: could not find a ready tiller pod
╷
│ Error: local-exec provisioner error
│
│ with null_resource.kube-cni,
│ on 03-kube-post-init.tf line 59, in resource "null_resource" "kube-cni":
│ 59: provisioner "local-exec" {
│
│ Error running command 'KUBECONFIG=secrets/admin.conf helm install -n kube-system hcloud-csi-driver mlohr/hcloud-csi-driver --set csiDriver.secret.create=true --set
│ csiDriver.secret.hcloudApiToken=0lQ5BEtHPUxodken3TCqF6pR7ZA112DSFmf5K71mEqM9YVUOSIiOj8Kt68LNM2bV': exit status 1. Output: Error: could not find a ready tiller pod
│
╵
I am trying to deploy 3 masters, 2 workers clusters and looks like some issue is happening with CNI and CoreDNS pods:
| => KUBECONFIG=secrets/admin.conf kubectl get nodes
KUBECONFIG=secrets/admin.conf kubectl get pods -A -o wide
NAME STATUS ROLES AGE VERSION
k8s-helsinki-master-1 NotReady master 10m v1.18.6
k8s-helsinki-master-2 NotReady master 8m52s v1.18.6
k8s-helsinki-master-3 NotReady master 7m41s v1.18.6
k8s-helsinki-node-1 NotReady <none> 5m53s v1.18.6
k8s-helsinki-node-2 NotReady <none> 6m13s v1.18.6
________________________________________________________________________________
| ~/Documents/Code/ubloquity/terraform-k8s-hetzner-DigitalOcean-Federation/hetzner_01 @ jperez-mbp (jperez)
| => KUBECONFIG=secrets/admin.conf kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-66bff467f8-9rj9r 0/1 Pending 0 10m <none> <none> <none> <none>
kube-system coredns-66bff467f8-qqzvp 0/1 Pending 0 10m <none> <none> <none> <none>
kube-system etcd-k8s-helsinki-master-1 1/1 Running 0 10m 65.21.0.135 k8s-helsinki-master-1 <none> <none>
kube-system etcd-k8s-helsinki-master-2 1/1 Running 0 8m48s 65.21.251.220 k8s-helsinki-master-2 <none> <none>
kube-system etcd-k8s-helsinki-master-3 1/1 Running 0 7m37s 65.21.4.190 k8s-helsinki-master-3 <none> <none>
kube-system kube-apiserver-k8s-helsinki-master-1 1/1 Running 0 10m 65.21.0.135 k8s-helsinki-master-1 <none> <none>
kube-system kube-apiserver-k8s-helsinki-master-2 1/1 Running 0 8m51s 65.21.251.220 k8s-helsinki-master-2 <none> <none>
kube-system kube-apiserver-k8s-helsinki-master-3 1/1 Running 0 7m40s 65.21.4.190 k8s-helsinki-master-3 <none> <none>
kube-system kube-controller-manager-k8s-helsinki-master-1 1/1 Running 1 10m 65.21.0.135 k8s-helsinki-master-1 <none> <none>
kube-system kube-controller-manager-k8s-helsinki-master-2 1/1 Running 0 8m51s 65.21.251.220 k8s-helsinki-master-2 <none> <none>
kube-system kube-controller-manager-k8s-helsinki-master-3 1/1 Running 0 7m41s 65.21.4.190 k8s-helsinki-master-3 <none> <none>
kube-system kube-proxy-6mhh7 1/1 Running 0 10m 65.21.0.135 k8s-helsinki-master-1 <none> <none>
kube-system kube-proxy-fxmhr 1/1 Running 0 7m42s 65.21.4.190 k8s-helsinki-master-3 <none> <none>
kube-system kube-proxy-h4lt9 1/1 Running 0 5m54s 65.21.251.5 k8s-helsinki-node-1 <none> <none>
kube-system kube-proxy-r85mj 1/1 Running 0 8m52s 65.21.251.220 k8s-helsinki-master-2 <none> <none>
kube-system kube-proxy-v2fvk 1/1 Running 0 6m14s 65.108.86.224 k8s-helsinki-node-2 <none> <none>
kube-system kube-scheduler-k8s-helsinki-master-1 1/1 Running 1 10m 65.21.0.135 k8s-helsinki-master-1 <none> <none>
kube-system kube-scheduler-k8s-helsinki-master-2 1/1 Running 0 8m52s 65.21.251.220 k8s-helsinki-master-2 <none> <none>
kube-system kube-scheduler-k8s-helsinki-master-3 1/1 Running 0 7m40s 65.21.4.190 k8s-helsinki-master-3 <none> <none>
kube-system tiller-deploy-56b574c76d-5t8bs 0/1 Pending 0 5m43s <none> <none> <none> <none>
kube-system tiller-deploy-587d84cd48-jl9nl 0/1 Pending 0 5m47s <none> <none> <none> <none>
Any ideas?
| => KUBECONFIG=secrets/admin.conf kubectl get pods --namespace=kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-66bff467f8-9rj9r 0/1 Pending 0 11m <none> <none> <none> <none>
coredns-66bff467f8-qqzvp 0/1 Pending 0 11m <none> <none> <none> <none>
etcd-k8s-helsinki-master-1 1/1 Running 0 11m 65.21.0.135 k8s-helsinki-master-1 <none> <none>
etcd-k8s-helsinki-master-2 1/1 Running 0 10m 65.21.251.220 k8s-helsinki-master-2 <none> <none>
etcd-k8s-helsinki-master-3 1/1 Running 0 9m1s 65.21.4.190 k8s-helsinki-master-3 <none> <none>
kube-apiserver-k8s-helsinki-master-1 1/1 Running 0 11m 65.21.0.135 k8s-helsinki-master-1 <none> <none>
kube-apiserver-k8s-helsinki-master-2 1/1 Running 0 10m 65.21.251.220 k8s-helsinki-master-2 <none> <none>
kube-apiserver-k8s-helsinki-master-3 1/1 Running 0 9m4s 65.21.4.190 k8s-helsinki-master-3 <none> <none>
kube-controller-manager-k8s-helsinki-master-1 1/1 Running 1 11m 65.21.0.135 k8s-helsinki-master-1 <none> <none>
kube-controller-manager-k8s-helsinki-master-2 1/1 Running 0 10m 65.21.251.220 k8s-helsinki-master-2 <none> <none>
kube-controller-manager-k8s-helsinki-master-3 1/1 Running 0 9m5s 65.21.4.190 k8s-helsinki-master-3 <none> <none>
kube-proxy-6mhh7 1/1 Running 0 11m 65.21.0.135 k8s-helsinki-master-1 <none> <none>
kube-proxy-fxmhr 1/1 Running 0 9m6s 65.21.4.190 k8s-helsinki-master-3 <none> <none>
kube-proxy-h4lt9 1/1 Running 0 7m18s 65.21.251.5 k8s-helsinki-node-1 <none> <none>
kube-proxy-r85mj 1/1 Running 0 10m 65.21.251.220 k8s-helsinki-master-2 <none> <none>
kube-proxy-v2fvk 1/1 Running 0 7m38s 65.108.86.224 k8s-helsinki-node-2 <none> <none>
kube-scheduler-k8s-helsinki-master-1 1/1 Running 1 11m 65.21.0.135 k8s-helsinki-master-1 <none> <none>
kube-scheduler-k8s-helsinki-master-2 1/1 Running 0 10m 65.21.251.220 k8s-helsinki-master-2 <none> <none>
kube-scheduler-k8s-helsinki-master-3 1/1 Running 0 9m4s 65.21.4.190 k8s-helsinki-master-3 <none> <none>
tiller-deploy-56b574c76d-5t8bs 0/1 Pending 0 7m7s <none> <none> <none> <none>
tiller-deploy-587d84cd48-jl9nl 0/1 Pending 0 7m11s <none> <none> <none> <none>
Looks like following some links:
coredns-pod-is-not-running-in-kubernetes?
| => KUBECONFIG=secrets/admin.conf kubectl describe pods coredns-66bff467f8-9rj9r --namespace=kube-system
Name: coredns-66bff467f8-9rj9r
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: <none>
Labels: k8s-app=kube-dns
pod-template-hash=66bff467f8
Annotations: <none>
Status: Pending
IP:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 11m default-scheduler 0/2 nodes are available: 2 node(s) had taint {node.cloudprovider.kubernetes.io/uninitialized: true}, that the pod didn't tolerate.
Warning FailedScheduling 10m default-scheduler 0/3 nodes are available: 3 node(s) had taint {node.cloudprovider.kubernetes.io/uninitialized: true}, that the pod didn't tolerate.
Warning FailedScheduling 9m7s default-scheduler 0/5 nodes are available: 5 node(s) had taint {node.cloudprovider.kubernetes.io/uninitialized: true}, that the pod didn't tolerate.
Warning FailedScheduling 9m7s default-scheduler 0/5 nodes are available: 5 node(s) had taint {node.cloudprovider.kubernetes.io/uninitialized: true}, that the pod didn't tolerate.
Warning FailedScheduling 12m (x3 over 13m) default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.cloudprovider.kubernetes.io/uninitialized: true}, that the pod didn't tolerate.
Warning FailedScheduling 12m (x3 over 12m) default-scheduler 0/2 nodes are available: 2 node(s) had taint {node.cloudprovider.kubernetes.io/uninitialized: true}, that the pod didn't tolerate.
yoplait commented
Basically this doesn't work anymore with the status on the hcloud-cloud-controller-manager:
| => KUBECONFIG=secrets/admin.conf kubectl describe nodes | egrep "Taints:|Name:"
Name: k8s-helsinki-master-1
Taints: node-role.kubernetes.io/master:NoSchedule
Warning FailedToCreateRoute 55m route_controller Could not create route 10893038-cf0b-4461-ac46-40d70e587670 10.244.0.0/24 for node k8s-helsinki-master-1 after 235.061594ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: k8s-helsinki-master-1 hcops/AllServersCache.getCache: not found
All nodes are failing because of the network problem ....
KUBECONFIG=secrets/admin.conf kubectl get events -A -w --sort-by='{.lastTimestamp}'
default 3m55s Warning SyncLoadBalancerFailed service/tmp-web Error syncing load balancer: failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.ReconcileHCLBTargets: hcops/providerIDToServerID: missing prefix hcloud://:
default 66s Warning FailedToCreateRoute node/k8s-helsinki-node-1 (combined from similar events): Could not create route 2eea7cc6-31c2-4cc1-88f9-e893f841222f 10.244.3.0/24 for node k8s-helsinki-node-1 after 728.214426ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: k8s-helsinki-node-1 hcops/AllServersCache.getCache: not found
default 66s Warning FailedToCreateRoute node/k8s-helsinki-master-3 (combined from similar events): Could not create route a4508fa8-3ad1-4a64-be11-c182135f663d 10.244.2.0/24 for node k8s-helsinki-master-3 after 470.881203ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: k8s-helsinki-master-3 hcops/AllServersCache.getCache: not found
default 66s Warning FailedToCreateRoute node/k8s-helsinki-master-2 (combined from similar events): Could not create route 7c7e89ff-3721-470e-844f-551691779032 10.244.1.0/24 for node k8s-helsinki-master-2 after 249.027669ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: k8s-helsinki-master-2 hcops/AllServersCache.getCache: not found
default 55s Warning FailedToCreateRoute node/k8s-helsinki-master-1 (combined from similar events): Could not create route 10893038-cf0b-4461-ac46-40d70e587670 10.244.0.0/24 for node k8s-helsinki-master-1 after 970.318239ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: k8s-helsinki-master-1 hcops/AllServersCache.getCache: not found
default 55s Warning FailedToCreateRoute node/k8s-helsinki-node-2 (combined from similar events): Could not create route 43207003-9d37-4ba8-b7db-ccb3f6bfab61 10.244.4.0/24 for node k8s-helsinki-node-2 after 764.871302ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: k8s-helsinki-node-2 hcops/AllServersCache.getCache: not found
default 42s Warning FailedScheduling pod/tmp-web-5cf74bc8c8-jjrpv 0/5 nodes are available: 2 node(s) had taint {node.kubernetes.io/network-unavailable: }, that the pod didn't tolerate, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
default 12s Warning FailedScheduling pod/tmp-web-5cf74bc8c8-k8j7n 0/5 nodes are available: 2 node(s) had taint {node.kubernetes.io/network-unavailable: }, that the pod didn't tolerate, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.