Failed to Create taint for control-plane
0x1un opened this issue · 6 comments
0x1un commented
TASK [init : Install base application] *************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.101]
ok: [192.168.137.11]
ok: [192.168.137.12]
ok: [192.168.137.10]
TASK [init : Install base application] *************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.11]
ok: [192.168.137.12]
ok: [192.168.137.10]
ok: [192.168.137.101]
TASK [init : Install base application] *************************************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.11]
skipping: [192.168.137.12]
skipping: [192.168.137.10]
skipping: [192.168.137.101]
PLAY [Set Haproxy + Keepalived] ********************************************************************************************************************************************************************************************************************************************************************************************
PLAY [Set Etcd Cluster] ****************************************************************************************************************************************************************************************************************************************************************************************************
TASK [etcd : Create etcd user group] ***************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.11]
ok: [192.168.137.10]
ok: [192.168.137.12]
TASK [etcd : Create etcd user] *********************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.12]
ok: [192.168.137.11]
TASK [etcd : Download etcd] ************************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
TASK [etcd : Install etcd] *************************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.11] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcd'})
ok: [192.168.137.10] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcd'})
ok: [192.168.137.12] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcd'})
ok: [192.168.137.10] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcdctl'})
ok: [192.168.137.11] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcdctl'})
ok: [192.168.137.12] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcdctl'})
ok: [192.168.137.10] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcdutl'})
ok: [192.168.137.12] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcdutl'})
ok: [192.168.137.11] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcdutl'})
TASK [etcd : Distribution certs] *******************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10] => (item={'line': '/opt/certs/etcd-ca.pem'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/etcd-ca.pem'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/etcd-ca.pem'})
ok: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/etcd-server.pem'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/etcd-server.pem'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/etcd-server.pem'})
ok: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/etcd-server.key'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/etcd-server.key'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/etcd-server.key'})
ok: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/etcd-peer.pem'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/etcd-peer.pem'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/etcd-peer.pem'})
ok: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/etcd-peer.key'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/etcd-peer.key'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/etcd-peer.key'})
ok: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/etcd-healthcheck-client.pem'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/etcd-healthcheck-client.pem'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/etcd-healthcheck-client.pem'})
ok: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/etcd-healthcheck-client.key'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/etcd-healthcheck-client.key'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/etcd-healthcheck-client.key'})
TASK [etcd : Create etcd data directory] ***********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
TASK [etcd : Generate etcd configure] **************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
PLAY [Set Control Plane Nodes] *********************************************************************************************************************************************************************************************************************************************************************************************
TASK [master : Create kubernetes user group] *******************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
TASK [master : Create kubernetes user] *************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
TASK [master : Create kubernetes config directory] *************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10] => (item=~/.kube)
ok: [192.168.137.11] => (item=~/.kube)
ok: [192.168.137.12] => (item=~/.kube)
ok: [192.168.137.10] => (item=/etc/kubernetes/pki)
ok: [192.168.137.11] => (item=/etc/kubernetes/pki)
ok: [192.168.137.12] => (item=/etc/kubernetes/pki)
TASK [master : Create kubernetes log directory] ****************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10] => (item=/var/log/kubernetes)
ok: [192.168.137.11] => (item=/var/log/kubernetes)
ok: [192.168.137.12] => (item=/var/log/kubernetes)
ok: [192.168.137.10] => (item=/usr/libexec/kubernetes)
ok: [192.168.137.11] => (item=/usr/libexec/kubernetes)
ok: [192.168.137.12] => (item=/usr/libexec/kubernetes)
TASK [master : Install kubectl] ********************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
TASK [master : Install master] *********************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-apiserver'})
changed: [192.168.137.10] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-apiserver'})
changed: [192.168.137.12] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-apiserver'})
changed: [192.168.137.11] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-controller-manager'})
changed: [192.168.137.12] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-controller-manager'})
changed: [192.168.137.10] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-controller-manager'})
changed: [192.168.137.11] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-scheduler'})
changed: [192.168.137.10] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-scheduler'})
changed: [192.168.137.12] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-scheduler'})
TASK [master : Distribution master certs] **********************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/sa.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/sa.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/sa.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/sa.pub'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/sa.pub'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/sa.pub'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/etcd-ca.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/etcd-ca.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/etcd-ca.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/apiserver.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/apiserver.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/apiserver.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/apiserver.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/apiserver.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/apiserver.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/apiserver-etcd-client.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/apiserver-etcd-client.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/apiserver-etcd-client.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/apiserver-etcd-client.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/apiserver-etcd-client.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/apiserver-etcd-client.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/apiserver-kubelet-client.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/apiserver-kubelet-client.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/apiserver-kubelet-client.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/apiserver-kubelet-client.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/apiserver-kubelet-client.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/apiserver-kubelet-client.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/front-proxy-ca.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/front-proxy-ca.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/front-proxy-ca.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/front-proxy-ca.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/front-proxy-ca.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/front-proxy-ca.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/front-proxy-client.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/front-proxy-client.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/front-proxy-client.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/front-proxy-client.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/front-proxy-client.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/front-proxy-client.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/controller-manager.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/controller-manager.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/controller-manager.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/controller-manager.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/controller-manager.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/controller-manager.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/scheduler.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/scheduler.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/scheduler.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/scheduler.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/scheduler.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/scheduler.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/admin.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/admin.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/admin.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/admin.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/admin.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/admin.pem'})
TASK [master : Get token-id] ***********************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.11]
TASK [master : Distribution kubectl kubeconfig] ****************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.11]
changed: [192.168.137.12]
TASK [master : Distribution master kubeconfig] *****************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10] => (item={'src': 'controller-manager.kubeconfig.j2', 'dest': '/etc/kubernetes/controller-manager.kubeconfig'})
changed: [192.168.137.11] => (item={'src': 'controller-manager.kubeconfig.j2', 'dest': '/etc/kubernetes/controller-manager.kubeconfig'})
changed: [192.168.137.12] => (item={'src': 'controller-manager.kubeconfig.j2', 'dest': '/etc/kubernetes/controller-manager.kubeconfig'})
changed: [192.168.137.10] => (item={'src': 'scheduler.kubeconfig.j2', 'dest': '/etc/kubernetes/scheduler.kubeconfig'})
changed: [192.168.137.11] => (item={'src': 'scheduler.kubeconfig.j2', 'dest': '/etc/kubernetes/scheduler.kubeconfig'})
changed: [192.168.137.12] => (item={'src': 'scheduler.kubeconfig.j2', 'dest': '/etc/kubernetes/scheduler.kubeconfig'})
TASK [master : Distribution master config] *********************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10] => (item={'src': 'kube-apiserver.conf.j2', 'dest': '/etc/kubernetes/kube-apiserver.conf'})
changed: [192.168.137.12] => (item={'src': 'kube-apiserver.conf.j2', 'dest': '/etc/kubernetes/kube-apiserver.conf'})
changed: [192.168.137.11] => (item={'src': 'kube-apiserver.conf.j2', 'dest': '/etc/kubernetes/kube-apiserver.conf'})
changed: [192.168.137.10] => (item={'src': 'kube-controller-manager.conf.j2', 'dest': '/etc/kubernetes/kube-controller-manager.conf'})
changed: [192.168.137.12] => (item={'src': 'kube-controller-manager.conf.j2', 'dest': '/etc/kubernetes/kube-controller-manager.conf'})
changed: [192.168.137.11] => (item={'src': 'kube-controller-manager.conf.j2', 'dest': '/etc/kubernetes/kube-controller-manager.conf'})
changed: [192.168.137.10] => (item={'src': 'kube-scheduler.conf.j2', 'dest': '/etc/kubernetes/kube-scheduler.conf'})
changed: [192.168.137.11] => (item={'src': 'kube-scheduler.conf.j2', 'dest': '/etc/kubernetes/kube-scheduler.conf'})
changed: [192.168.137.12] => (item={'src': 'kube-scheduler.conf.j2', 'dest': '/etc/kubernetes/kube-scheduler.conf'})
changed: [192.168.137.10] => (item={'src': 'audit-policy.yaml.j2', 'dest': '/etc/kubernetes/audit-policy.yaml'})
changed: [192.168.137.11] => (item={'src': 'audit-policy.yaml.j2', 'dest': '/etc/kubernetes/audit-policy.yaml'})
changed: [192.168.137.12] => (item={'src': 'audit-policy.yaml.j2', 'dest': '/etc/kubernetes/audit-policy.yaml'})
TASK [master : Distribution master systemd unit] ***************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10] => (item={'src': 'kube-apiserver.service.j2', 'dest': '/usr/lib/systemd/system/kube-apiserver.service'})
changed: [192.168.137.12] => (item={'src': 'kube-apiserver.service.j2', 'dest': '/usr/lib/systemd/system/kube-apiserver.service'})
changed: [192.168.137.11] => (item={'src': 'kube-apiserver.service.j2', 'dest': '/usr/lib/systemd/system/kube-apiserver.service'})
changed: [192.168.137.10] => (item={'src': 'kube-controller-manager.service.j2', 'dest': '/usr/lib/systemd/system/kube-controller-manager.service'})
changed: [192.168.137.12] => (item={'src': 'kube-controller-manager.service.j2', 'dest': '/usr/lib/systemd/system/kube-controller-manager.service'})
changed: [192.168.137.11] => (item={'src': 'kube-controller-manager.service.j2', 'dest': '/usr/lib/systemd/system/kube-controller-manager.service'})
changed: [192.168.137.10] => (item={'src': 'kube-scheduler.service.j2', 'dest': '/usr/lib/systemd/system/kube-scheduler.service'})
changed: [192.168.137.12] => (item={'src': 'kube-scheduler.service.j2', 'dest': '/usr/lib/systemd/system/kube-scheduler.service'})
changed: [192.168.137.11] => (item={'src': 'kube-scheduler.service.j2', 'dest': '/usr/lib/systemd/system/kube-scheduler.service'})
TASK [master : Restart kube-apiserver] *************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.10]
changed: [192.168.137.12]
TASK [master : Restart kube-controller-manager] ****************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.10]
changed: [192.168.137.12]
TASK [master : Restart kube-scheduler] *************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.12]
changed: [192.168.137.10]
changed: [192.168.137.11]
TASK [master : Add kubectl completion] *************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.11]
TASK [master : Waiting kube-apiserver starting] ****************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.12]
ok: [192.168.137.11]
TASK [master : Waiting kube-controller-manager starting] *******************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
TASK [master : Waiting kube-scheduler starting] ****************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
TASK [master : Kube-apiserver health check] ********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.12]
ok: [192.168.137.11]
ok: [192.168.137.10]
TASK [master : Kube-controller-manager health check] ***********************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
TASK [master : Kube-scheduler health check] ********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.12]
ok: [192.168.137.11]
PLAY [Set Container Runtime] ***********************************************************************************************************************************************************************************************************************************************************************************************
TASK [containerd : Install dependency] *************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.101]
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.11]
TASK [containerd : Install dependency] *************************************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.10]
skipping: [192.168.137.11]
skipping: [192.168.137.12]
skipping: [192.168.137.101]
TASK [containerd : Add nvidia-container-toolkit repository] ****************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.10] => (item={'name': 'libnvidia-container', 'description': 'libnvidia-container', 'baseurl': 'https://nvidia.github.io/libnvidia-container/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/libnvidia-container/gpgkey'})
skipping: [192.168.137.10] => (item={'name': 'nvidia-container-runtime', 'description': 'nvidia-container-runtime', 'baseurl': 'https://nvidia.github.io/nvidia-container-runtime/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/nvidia-container-runtime/gpgkey'})
skipping: [192.168.137.11] => (item={'name': 'libnvidia-container', 'description': 'libnvidia-container', 'baseurl': 'https://nvidia.github.io/libnvidia-container/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/libnvidia-container/gpgkey'})
skipping: [192.168.137.11] => (item={'name': 'nvidia-container-runtime', 'description': 'nvidia-container-runtime', 'baseurl': 'https://nvidia.github.io/nvidia-container-runtime/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/nvidia-container-runtime/gpgkey'})
skipping: [192.168.137.12] => (item={'name': 'libnvidia-container', 'description': 'libnvidia-container', 'baseurl': 'https://nvidia.github.io/libnvidia-container/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/libnvidia-container/gpgkey'})
skipping: [192.168.137.12] => (item={'name': 'nvidia-container-runtime', 'description': 'nvidia-container-runtime', 'baseurl': 'https://nvidia.github.io/nvidia-container-runtime/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/nvidia-container-runtime/gpgkey'})
skipping: [192.168.137.101] => (item={'name': 'libnvidia-container', 'description': 'libnvidia-container', 'baseurl': 'https://nvidia.github.io/libnvidia-container/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/libnvidia-container/gpgkey'})
skipping: [192.168.137.101] => (item={'name': 'nvidia-container-runtime', 'description': 'nvidia-container-runtime', 'baseurl': 'https://nvidia.github.io/nvidia-container-runtime/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/nvidia-container-runtime/gpgkey'})
TASK [containerd : Add nvidia-container-toolkit repository] ****************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.10]
skipping: [192.168.137.11]
skipping: [192.168.137.12]
skipping: [192.168.137.101]
TASK [containerd : Install nvidia-container-runtime] ***********************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.10]
skipping: [192.168.137.11]
skipping: [192.168.137.12]
skipping: [192.168.137.101]
TASK [containerd : Install runc] *******************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.101]
changed: [192.168.137.11]
changed: [192.168.137.10]
changed: [192.168.137.12]
TASK [containerd : Create cni directory] ***********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
ok: [192.168.137.101]
TASK [containerd : Install cni] ********************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.12]
changed: [192.168.137.10]
changed: [192.168.137.11]
changed: [192.168.137.101]
TASK [containerd : Install containerd] *************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.10]
changed: [192.168.137.101]
TASK [containerd : Set containerd service] *********************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.101]
TASK [containerd : Create containerd data directory] ***********************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.101]
TASK [containerd : Create containerd config directory] *********************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
ok: [192.168.137.101]
TASK [containerd : Set containerd config] **********************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.10]
changed: [192.168.137.101]
changed: [192.168.137.12]
TASK [containerd : Install cri-tools] **************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.101]
changed: [192.168.137.11]
TASK [containerd : Config cri-tools] ***************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.101]
RUNNING HANDLER [containerd : restart containerd] **************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.10]
changed: [192.168.137.101]
RUNNING HANDLER [containerd : Containerd | restart containerd] *************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.101]
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.10]
RUNNING HANDLER [containerd : Containerd | wait for containerd] ************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.101]
changed: [192.168.137.10]
RUNNING HANDLER [containerd : Get crictl completion] ***********************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.12]
ok: [192.168.137.101]
ok: [192.168.137.11]
RUNNING HANDLER [containerd : Install crictl completion] *******************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.101]
changed: [192.168.137.11]
PLAY [Set Worker Nodes] ****************************************************************************************************************************************************************************************************************************************************************************************************
TASK [worker : Install dependency] *****************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
ok: [192.168.137.101]
TASK [worker : Install dependency] *****************************************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.10]
skipping: [192.168.137.11]
skipping: [192.168.137.12]
skipping: [192.168.137.101]
TASK [worker : Create kubernetes directory] ********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10] => (item={'line': '/etc/kubernetes/pki'})
ok: [192.168.137.11] => (item={'line': '/etc/kubernetes/pki'})
ok: [192.168.137.12] => (item={'line': '/etc/kubernetes/pki'})
changed: [192.168.137.101] => (item={'line': '/etc/kubernetes/pki'})
changed: [192.168.137.10] => (item={'line': '/etc/kubernetes/manifests'})
changed: [192.168.137.11] => (item={'line': '/etc/kubernetes/manifests'})
changed: [192.168.137.12] => (item={'line': '/etc/kubernetes/manifests'})
changed: [192.168.137.101] => (item={'line': '/etc/kubernetes/manifests'})
ok: [192.168.137.10] => (item={'line': '/var/lib/kubelet'})
ok: [192.168.137.11] => (item={'line': '/var/lib/kubelet'})
ok: [192.168.137.101] => (item={'line': '/var/lib/kubelet'})
ok: [192.168.137.12] => (item={'line': '/var/lib/kubelet'})
TASK [worker : Create kubelet directory] ***********************************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.10]
skipping: [192.168.137.11]
skipping: [192.168.137.12]
skipping: [192.168.137.101]
TASK [worker : Install worker node] ****************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.12] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kubelet'})
changed: [192.168.137.11] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kubelet'})
changed: [192.168.137.10] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kubelet'})
changed: [192.168.137.101] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kubelet'})
changed: [192.168.137.12] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-proxy'})
changed: [192.168.137.11] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-proxy'})
changed: [192.168.137.10] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-proxy'})
changed: [192.168.137.101] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-proxy'})
TASK [worker : Distribution worker certs] **********************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.101] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.101] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.101] => (item={'line': '/opt/certs/192.168.137.101/kube-proxy.pem'})
changed: [192.168.137.101] => (item={'line': '/opt/certs/192.168.137.101/kube-proxy.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/kube-proxy.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/kube-proxy.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/kube-proxy.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/kube-proxy.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/kube-proxy.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/kube-proxy.key'})
TASK [worker : Get bootstrap-token-id] *************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.101]
TASK [worker : Get bootstrap-token-secret] *********************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.101]
changed: [192.168.137.11]
TASK [worker : Distribution worker kubeconfig] *****************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10] => (item={'src': 'bootstrap.kubeconfig.j2', 'dest': '/etc/kubernetes/bootstrap.kubeconfig'})
changed: [192.168.137.11] => (item={'src': 'bootstrap.kubeconfig.j2', 'dest': '/etc/kubernetes/bootstrap.kubeconfig'})
changed: [192.168.137.12] => (item={'src': 'bootstrap.kubeconfig.j2', 'dest': '/etc/kubernetes/bootstrap.kubeconfig'})
changed: [192.168.137.101] => (item={'src': 'bootstrap.kubeconfig.j2', 'dest': '/etc/kubernetes/bootstrap.kubeconfig'})
changed: [192.168.137.10] => (item={'src': 'proxy.kubeconfig.j2', 'dest': '/etc/kubernetes/proxy.kubeconfig'})
changed: [192.168.137.11] => (item={'src': 'proxy.kubeconfig.j2', 'dest': '/etc/kubernetes/proxy.kubeconfig'})
changed: [192.168.137.101] => (item={'src': 'proxy.kubeconfig.j2', 'dest': '/etc/kubernetes/proxy.kubeconfig'})
changed: [192.168.137.12] => (item={'src': 'proxy.kubeconfig.j2', 'dest': '/etc/kubernetes/proxy.kubeconfig'})
TASK [worker : Distribution worker config] *********************************************************************************************************************************************************************************************************************************************************************************
[DEPRECATION WARNING]: Use 'ansible.utils.next_nth_usable' module instead. This feature will be removed from ansible.netcommon in a release after 2024-01-01. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: Use 'ansible.utils.next_nth_usable' module instead. This feature will be removed from ansible.netcommon in a release after 2024-01-01. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: Use 'ansible.utils.next_nth_usable' module instead. This feature will be removed from ansible.netcommon in a release after 2024-01-01. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: Use 'ansible.utils.next_nth_usable' module instead. This feature will be removed from ansible.netcommon in a release after 2024-01-01. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
changed: [192.168.137.10] => (item={'src': 'kubelet.conf.j2', 'dest': '/etc/kubernetes/kubelet.conf'})
changed: [192.168.137.11] => (item={'src': 'kubelet.conf.j2', 'dest': '/etc/kubernetes/kubelet.conf'})
changed: [192.168.137.12] => (item={'src': 'kubelet.conf.j2', 'dest': '/etc/kubernetes/kubelet.conf'})
changed: [192.168.137.101] => (item={'src': 'kubelet.conf.j2', 'dest': '/etc/kubernetes/kubelet.conf'})
changed: [192.168.137.10] => (item={'src': '10-kubelet.conf.j2', 'dest': '/etc/sysconfig/kubelet'})
changed: [192.168.137.11] => (item={'src': '10-kubelet.conf.j2', 'dest': '/etc/sysconfig/kubelet'})
changed: [192.168.137.12] => (item={'src': '10-kubelet.conf.j2', 'dest': '/etc/sysconfig/kubelet'})
changed: [192.168.137.101] => (item={'src': '10-kubelet.conf.j2', 'dest': '/etc/sysconfig/kubelet'})
changed: [192.168.137.10] => (item={'src': 'kube-proxy.conf.j2', 'dest': '/etc/kubernetes/kube-proxy.conf'})
changed: [192.168.137.11] => (item={'src': 'kube-proxy.conf.j2', 'dest': '/etc/kubernetes/kube-proxy.conf'})
changed: [192.168.137.101] => (item={'src': 'kube-proxy.conf.j2', 'dest': '/etc/kubernetes/kube-proxy.conf'})
changed: [192.168.137.12] => (item={'src': 'kube-proxy.conf.j2', 'dest': '/etc/kubernetes/kube-proxy.conf'})
TASK [worker : Distribution worker system unit] ****************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10] => (item={'src': 'kubelet.service.j2', 'dest': '/usr/lib/systemd/system/kubelet.service'})
changed: [192.168.137.11] => (item={'src': 'kubelet.service.j2', 'dest': '/usr/lib/systemd/system/kubelet.service'})
changed: [192.168.137.12] => (item={'src': 'kubelet.service.j2', 'dest': '/usr/lib/systemd/system/kubelet.service'})
changed: [192.168.137.101] => (item={'src': 'kubelet.service.j2', 'dest': '/usr/lib/systemd/system/kubelet.service'})
changed: [192.168.137.10] => (item={'src': 'kube-proxy.service.j2', 'dest': '/usr/lib/systemd/system/kube-proxy.service'})
changed: [192.168.137.11] => (item={'src': 'kube-proxy.service.j2', 'dest': '/usr/lib/systemd/system/kube-proxy.service'})
changed: [192.168.137.12] => (item={'src': 'kube-proxy.service.j2', 'dest': '/usr/lib/systemd/system/kube-proxy.service'})
changed: [192.168.137.101] => (item={'src': 'kube-proxy.service.j2', 'dest': '/usr/lib/systemd/system/kube-proxy.service'})
TASK [worker : Check if bootstrap-token exists] ****************************************************************************************************************************************************************************************************************************************************************************
fatal: [192.168.137.10]: FAILED! => {"changed": true, "cmd": "kubectl -n kube-system get secret bootstrap-token-4febb9", "delta": "0:00:00.712825", "end": "2023-02-07 16:12:57.668711", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:12:56.955886", "stderr": "Error from server (NotFound): secrets \"bootstrap-token-4febb9\" not found", "stderr_lines": ["Error from server (NotFound): secrets \"bootstrap-token-4febb9\" not found"], "stdout": "", "stdout_lines": []}
...ignoring
TASK [worker : Create bootstrap-token secret] ******************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
TASK [worker : Check if clusterrolebinding kubelet-bootstrap exists] *******************************************************************************************************************************************************************************************************************************************************
fatal: [192.168.137.10]: FAILED! => {"changed": true, "cmd": "kubectl get clusterrolebinding kubelet-bootstrap", "delta": "0:00:00.606823", "end": "2023-02-07 16:13:00.005821", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:12:59.398998", "stderr": "Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"kubelet-bootstrap\" not found", "stderr_lines": ["Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"kubelet-bootstrap\" not found"], "stdout": "", "stdout_lines": []}
...ignoring
TASK [worker : Create clusterrolebinding kubelet-bootstrap] ****************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
TASK [worker : Check if node-autoapprove-bootstrap exists] *****************************************************************************************************************************************************************************************************************************************************************
fatal: [192.168.137.10]: FAILED! => {"changed": true, "cmd": "kubectl get clusterrolebinding node-autoapprove-bootstrap", "delta": "0:00:00.609772", "end": "2023-02-07 16:13:02.301013", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:13:01.691241", "stderr": "Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"node-autoapprove-bootstrap\" not found", "stderr_lines": ["Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"node-autoapprove-bootstrap\" not found"], "stdout": "", "stdout_lines": []}
...ignoring
TASK [worker : Create clusterrolebinding node-autoapprove-bootstrap] *******************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
TASK [worker : Check if clusterrolebinding node-autoapprove-certificate-rotation exists] ***********************************************************************************************************************************************************************************************************************************
fatal: [192.168.137.10]: FAILED! => {"changed": true, "cmd": "kubectl get clusterrolebinding node-autoapprove-certificate-rotation", "delta": "0:00:00.630910", "end": "2023-02-07 16:13:04.735579", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:13:04.104669", "stderr": "Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"node-autoapprove-certificate-rotation\" not found", "stderr_lines": ["Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"node-autoapprove-certificate-rotation\" not found"], "stdout": "", "stdout_lines": []}
...ignoring
TASK [worker : Create clusterrolebinding node-autoapprove-certificate-rotation] ********************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
TASK [worker : Restart kubelet] ********************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.101]
changed: [192.168.137.11]
changed: [192.168.137.12]
TASK [worker : Waiting kubelet starting] ***********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.101]
ok: [192.168.137.12]
TASK [worker : kubelet health check] ***************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.12]
ok: [192.168.137.11]
ok: [192.168.137.10]
ok: [192.168.137.101]
TASK [worker : Restart kube-proxy] *****************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.11]
changed: [192.168.137.101]
changed: [192.168.137.12]
TASK [worker : Waiting kube-proxy starting] ********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
ok: [192.168.137.101]
TASK [worker : kube-proxy health check] ************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.101]
ok: [192.168.137.12]
TASK [worker : Create taint for control-plane] *****************************************************************************************************************************************************************************************************************************************************************************
failed: [192.168.137.10] (item=192.168.137.10) => {"ansible_loop_var": "item", "changed": true, "cmd": "kubectl taint nodes master-01-192.168.137.10 node-role.kubernetes.io/control-plane=:NoSchedule --overwrite", "delta": "0:00:00.610875", "end": "2023-02-07 16:16:03.459949", "item": "192.168.137.10", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:16:02.849074", "stderr": "Error from server (NotFound): nodes \"master-01-192.168.137.10\" not found", "stderr_lines": ["Error from server (NotFound): nodes \"master-01-192.168.137.10\" not found"], "stdout": "", "stdout_lines": []}
failed: [192.168.137.10] (item=192.168.137.11) => {"ansible_loop_var": "item", "changed": true, "cmd": "kubectl taint nodes master-02-192.168.137.11 node-role.kubernetes.io/control-plane=:NoSchedule --overwrite", "delta": "0:00:00.729596", "end": "2023-02-07 16:16:04.576235", "item": "192.168.137.11", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:16:03.846639", "stderr": "Error from server (NotFound): nodes \"master-02-192.168.137.11\" not found", "stderr_lines": ["Error from server (NotFound): nodes \"master-02-192.168.137.11\" not found"], "stdout": "", "stdout_lines": []}
failed: [192.168.137.10] (item=192.168.137.12) => {"ansible_loop_var": "item", "changed": true, "cmd": "kubectl taint nodes master-03-192.168.137.12 node-role.kubernetes.io/control-plane=:NoSchedule --overwrite", "delta": "0:00:00.628364", "end": "2023-02-07 16:16:05.655418", "item": "192.168.137.12", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:16:05.027054", "stderr": "Error from server (NotFound): nodes \"master-03-192.168.137.12\" not found", "stderr_lines": ["Error from server (NotFound): nodes \"master-03-192.168.137.12\" not found"], "stdout": "", "stdout_lines": []}
NO MORE HOSTS LEFT *********************************************************************************************************************************************************************************************************************************************************************************************************
PLAY RECAP *****************************************************************************************************************************************************************************************************************************************************************************************************************
192.168.137.10 : ok=97 changed=44 unreachable=0 failed=1 skipped=12 rescued=0 ignored=4
192.168.137.101 : ok=59 changed=25 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0
192.168.137.11 : ok=93 changed=36 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0
192.168.137.12 : ok=89 changed=36 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0
localhost : ok=47 changed=2 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0
输出太长了，我截取了一部分。
kubeop commented
最终集群是否部署成功？如果集群成功，这个可能是节点注册太慢，节点还没成功注册，脚本已经执行了。我后续加个判断处理一下
0x1un commented
你好，这几天有点事情耽搁了，节点部署后 kubectl get nodes 没有节点加入进来。
0x1un commented
刚才我在本地虚拟机创建了一组测试集群，完全干净的centos7环境，完整跑了一遍最新的playbook。
这是我的inventory：
# 本组内填写etcd服务器及主机名(会校验是否以横杠分割),集群节点名称会截取主机名以横杠分割后两段
[etcd]
192.168.137.201 hostname=etcd-01 ansible_ssh_pass="123456"
[haproxy]
192.168.137.101 hostname=haproxy-01 type=BACKUP priority=90 ansible_ssh_pass="123456"
192.168.137.102 hostname=haproxy-02 type=MASTER priority=100 ansible_ssh_pass="123456"
# 本组内填写master服务器及主机名(会校验是否以横杠分割),集群节点名称会截取主机名以横杠分割后两段
[master]
192.168.137.10 hostname=master-01 ansible_ssh_pass="123456"
# 本组内填写worker服务器及主机名(会校验是否以横杠分割),集群节点名称会截取主机名以横杠分割后两段
# 最后面添加gpu=true 表示节点为GPU节点，运行时会配置使用GPU并且添加nvidia.com/gpu=true标签
# 不是GPU节点时，可去掉gpu配置项
# 启用GPU时，请先在节点按照 https://nvidia.github.io/libnvidia-container/ 配置软件源或同步相关包到私服
[worker]
192.168.137.11 hostname=worker-01 gpu=false ansible_ssh_pass="123456"
kubelet日志提示：
Feb 15 13:36:26 worker-01 kubelet[15536]: E0215 13:36:26.179937 15536 certificate_manager.go:471] kubernetes.io/kube-apiserver-client-kubelet: Failed while requesting a signed certificate from the control plane: cannot create certificate signing request: Post "https://172.16.90.100:6443/apis/certificates.k8s.io/v1/certificatesigningrequests": dial tcp 172.16.90.100:6443: connect: connection refused
我发现ip a中并没有创建172.16.90.x网段：
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:aa:a3:91 brd ff:ff:ff:ff:ff:ff
inet 192.168.137.11/24 brd 192.168.137.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::876f:be9e:31dd:d657/64 scope link noprefixroute
valid_lft forever preferred_lft forever
查看containerd日志提示： cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config
# systemctl status containerd -l
● containerd.service - containerd container runtime
Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2023-02-15 13:29:10 CST; 11min ago
Docs: https://containerd.io
Main PID: 18078 (containerd)
Tasks: 9
Memory: 22.3M
CGroup: /system.slice/containerd.service
└─18078 /usr/local/bin/containerd
Feb 15 13:29:10 master-01 containerd[18078]: time="2023-02-15T13:29:10.934210761+08:00" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Feb 15 13:29:10 master-01 containerd[18078]: time="2023-02-15T13:29:10.935368901+08:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Feb 15 13:29:10 master-01 containerd[18078]: time="2023-02-15T13:29:10.935427990+08:00" level=info msg=serving... address=/run/containerd/containerd.sock
Feb 15 13:29:10 master-01 containerd[18078]: time="2023-02-15T13:29:10.936674204+08:00" level=info msg="containerd successfully booted in 0.044534s"
Feb 15 13:29:10 master-01 containerd[18078]: time="2023-02-15T13:29:10.936722241+08:00" level=info msg="Start subscribing containerd event"
Feb 15 13:29:10 master-01 containerd[18078]: time="2023-02-15T13:29:10.936762161+08:00" level=info msg="Start recovering state"
Feb 15 13:29:10 master-01 containerd[18078]: time="2023-02-15T13:29:10.936882597+08:00" level=info msg="Start event monitor"
其中的playbook执行统计：
PLAY RECAP *********************************************************************************************************************************************************************************************************************************************************************************************
192.168.137.10 : ok=92 changed=73 unreachable=0 failed=1 skipped=13 rescued=0 ignored=4
192.168.137.101 : ok=39 changed=27 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0
192.168.137.102 : ok=35 changed=27 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0
192.168.137.11 : ok=62 changed=49 unreachable=0 failed=1 skipped=13 rescued=0 ignored=0
192.168.137.201 : ok=39 changed=29 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0
localhost : ok=60 changed=47 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
192.168.137.10、11 失败的部分：
TASK [worker : Check whether the worker is ready?] *****************************************************************************************************************************************************************************************************************************************************
FAILED - RETRYING: [192.168.137.10]: Check whether the worker is ready? (5 retries left).
FAILED - RETRYING: [192.168.137.11 -> 192.168.137.10]: Check whether the worker is ready? (5 retries left).
FAILED - RETRYING: [192.168.137.10]: Check whether the worker is ready? (4 retries left).
FAILED - RETRYING: [192.168.137.11 -> 192.168.137.10]: Check whether the worker is ready? (4 retries left).
FAILED - RETRYING: [192.168.137.10]: Check whether the worker is ready? (3 retries left).
FAILED - RETRYING: [192.168.137.11 -> 192.168.137.10]: Check whether the worker is ready? (3 retries left).
FAILED - RETRYING: [192.168.137.10]: Check whether the worker is ready? (2 retries left).
FAILED - RETRYING: [192.168.137.11 -> 192.168.137.10]: Check whether the worker is ready? (2 retries left).
FAILED - RETRYING: [192.168.137.10]: Check whether the worker is ready? (1 retries left).
FAILED - RETRYING: [192.168.137.11 -> 192.168.137.10]: Check whether the worker is ready? (1 retries left).
failed: [192.168.137.10] (item=192.168.137.10) => {"ansible_loop_var": "item", "attempts": 5, "changed": true, "cmd": "kubectl get node | grep master-01-192.168.137.10", "delta": "0:00:00.122645", "end": "2023-02-15 13:34:14.207115", "item": "192.168.137.10", "msg": "non-zero return code", "rc": 1, "start": "2023-02-15 13:34:14.084470", "stderr": "No resources found", "stderr_lines": ["No resources found"], "stdout": "", "stdout_lines": []}
failed: [192.168.137.11 -> 192.168.137.10] (item=192.168.137.10) => {"ansible_loop_var": "item", "attempts": 5, "changed": true, "cmd": "kubectl get node | grep worker-01-192.168.137.11", "delta": "0:00:00.084675", "end": "2023-02-15 13:34:14.201377", "item": "192.168.137.10", "msg": "non-zero return code", "rc": 1, "start": "2023-02-15 13:34:14.116702", "stderr": "No resources found", "stderr_lines": ["No resources found"], "stdout": "", "stdout_lines": []}
192.168.137.10 ignoring的部分：
TASK [worker : Check if bootstrap-token exists] ********************************************************************************************************************************************************************************************************************************************************
fatal: [192.168.137.10]: FAILED! => {"changed": true, "cmd": "kubectl -n kube-system get secret bootstrap-token-f24e1b", "delta": "0:00:00.130208", "end": "2023-02-15 13:31:02.274777", "msg": "non-zero return code", "rc": 1, "start": "2023-02-15 13:31:02.144569", "stderr": "Error from server (NotFound): secrets \"bootstrap-token-f24e1b\" not found", "stderr_lines": ["Error from server (NotFound): secrets \"bootstrap-token-f24e1b\" not found"], "stdout": "", "stdout_lines": []}
...ignoring
TASK [worker : Create bootstrap-token secret] **********************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
TASK [worker : Check if clusterrolebinding kubelet-bootstrap exists] ***********************************************************************************************************************************************************************************************************************************
fatal: [192.168.137.10]: FAILED! => {"changed": true, "cmd": "kubectl get clusterrolebinding kubelet-bootstrap", "delta": "0:00:00.062390", "end": "2023-02-15 13:31:03.230850", "msg": "non-zero return code", "rc": 1, "start": "2023-02-15 13:31:03.168460", "stderr": "Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"kubelet-bootstrap\" not found", "stderr_lines": ["Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"kubelet-bootstrap\" not found"], "stdout": "", "stdout_lines": []}
...ignoring
TASK [worker : Create clusterrolebinding kubelet-bootstrap] ********************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
TASK [worker : Check if node-autoapprove-bootstrap exists] *********************************************************************************************************************************************************************************************************************************************
fatal: [192.168.137.10]: FAILED! => {"changed": true, "cmd": "kubectl get clusterrolebinding node-autoapprove-bootstrap", "delta": "0:00:00.063945", "end": "2023-02-15 13:31:04.234837", "msg": "non-zero return code", "rc": 1, "start": "2023-02-15 13:31:04.170892", "stderr": "Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"node-autoapprove-bootstrap\" not found", "stderr_lines": ["Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"node-autoapprove-bootstrap\" not found"], "stdout": "", "stdout_lines": []}
...ignoring
TASK [worker : Create clusterrolebinding node-autoapprove-bootstrap] ***********************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
TASK [worker : Check if clusterrolebinding node-autoapprove-certificate-rotation exists] ***************************************************************************************************************************************************************************************************************
fatal: [192.168.137.10]: FAILED! => {"changed": true, "cmd": "kubectl get clusterrolebinding node-autoapprove-certificate-rotation", "delta": "0:00:00.111519", "end": "2023-02-15 13:31:05.209951", "msg": "non-zero return code", "rc": 1, "start": "2023-02-15 13:31:05.098432", "stderr": "Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"node-autoapprove-certificate-rotation\" not found", "stderr_lines": ["Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"node-autoapprove-certificate-rotation\" not found"], "stdout": "", "stdout_lines": []}
...ignoring
kubeop commented
group_vars/all.yml 中需要指定负载均衡的VIP和端口的
kubeop commented
如果你只有一个master节点，是可以跳过安装haproxy和keepalived的，group_vars/all.yml 中lb的ip和端口指定为这一台master节点的IP和apiserver的端口，然后执行以下命令即可。
ansible-playbook cluster.yml -i inventory --skip-tags=haproxy,keepalived
0x1un commented
多谢，清理了环境修改loadbalance.ip 重新跑了一遍解决了，大意了:)。