kube-spawn fails because of SSL errors
Closed this issue · 8 comments
alban commented
After working around issue #114, I tried again and got the following error:
+ systemctl start kubelet.service
+ kubeadm join --skip-preflight-checks --token b3ffe9.38cc5b6b05b0a4c8 10.22.0.5:6443
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Skipping pre-flight checks
[discovery] Trying to connect to API Server "10.22.0.5:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.22.0.5:6443"
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [Get https://10.22.0.5:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.22.0.5:6443: getsockopt: connection refused]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [Get https://10.22.0.5:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.22.0.5:6443: getsockopt: connection refused]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [Get https://10.22.0.5:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.22.0.5:6443: getsockopt: connection refused]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [Get https://10.22.0.5:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.22.0.5:6443: getsockopt: connection refused]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [the server has asked for the client to provide credentials (get configmaps cluster-info)]
On node 1 (sudo machinectl shell kube-spawn-1), I tried:
kube-spawn-1 ~ # curl https://10.22.0.5:6443
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.
dongsupark commented
First of all, firewall issues should be handled automatically, I suppose.
Port 6443 is supposed to be opened by kube-apiserver. Can you check if kube-apiserver is already running on kube-spawn-0?
If not, try to wait for several minutes until every pod became available, running kubectl get pods --all-namespaces -w?
I have no idea about the ssl issue, but apiserver already runs, the ssl issue could be also gone.
iaguis commented
I think the SSL issue is because @alban didn't pass --cacert to curl with the right certificate to connect to the API server, so probably not related.
alban commented
yes, kube-apiserver is running. And I can connect to ncat localhost 6443 from kube-spawn-0.
$ kubectl get nodes
NAME STATUS AGE VERSION
kube-spawn-0 Ready 5m v1.6.4
$ kubectl get pods
No resources found.
$ kubectl get pods --all-namespaces -w
The connection to the server 10.22.0.11:6443 was refused - did you specify the right host or port?
Connectivity breaks after a while.
alban commented
$ kubectl get pods --all-namespaces -w
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-kube-spawn-0 1/1 Running 0 14m
kube-system kube-apiserver-kube-spawn-0 1/1 Running 8 13m
kube-system kube-controller-manager-kube-spawn-0 1/1 Running 4 14m
kube-system kube-dns-3913472980-mvrwn 2/3 Running 11 13m
kube-system kube-proxy-cg5s7 0/1 CrashLoopBackOff 7 13m
kube-system kube-scheduler-kube-spawn-0 1/1 Running 4 14m
kube-system weave-net-rdkb0 2/2 Running 0 13m
kube-system kube-apiserver-kube-spawn-0 1/1 Running 9 14m
and then it stops again
alban commented
I had old kubectl and other tools in the k8s directory. After a sudo git clean -fdx, and rebuilding, it works better.
dongsupark commented
#118 was merged, v0.1.1 was tagged. When I tested it yesterday, I didn't see any error like described in this issue.
So I'll close it. Thanks!