vmware-tanzu/cluster-api-provider-bringyourownhost

kubeconfig generation CSR failure

danieljkemp opened this issue · 10 comments

danieljkemp commented

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

WHen running the BYOH agent on the new node, I am getting the following error

I1205 20:48:13.208247   36593 main.go:230]  "msg"="initiated bootstrap kubeconfig flow" 
I1205 20:48:13.213938   36593 csr.go:120]  "msg"="certTimeToExpire" "duration"=31536000000000000
E1205 20:48:13.226649   36593 csr.go:129]  "msg"="in request certificate" "error"="cannot create certificate signing request: Unauthorized" 
E1205 20:48:13.226731   36593 main.go:161]  "msg"="bootstrap flow failed" "error"="kubeconfig generation failed: cannot create certificate signing request: Unauthorized"

What did you expect to happen:
No errors, and the node visible in kubectl get byohosts

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

  • Cluster-api-provider-bringyourownhost version: 0.3.1
  • Kubernetes version: (use kubectl version --short): 1.24
  • OS (e.g. from /etc/os-release): Ubuntu 20.04.5 LTS (Focal Fossa)
dharmjit commented

Hi @danieljkemp, Thanks for trying out BYOH, This seems like an RBAC issue. Did you follow the steps in the getting started guide to create the bootstrap kubeconfig[here] for the initial one-time use in the host? This provides a bootstrap token kubeconfig with the required permissions to create CSR.

danieljkemp commented
FischerLGLN commented

Same error on k8s 1.25.4 bootstrap cluster. Has it something to do with service accounts missing secrets, thus kubeconfig being not valid anymore? I think this happens since 1.24+

FischerLGLN commented

Same error on k8s 1.23.5 bootstrap cluster unfortunately.

FischerLGLN commented

@danieljkemp
Okay, the error is that the wrong bootstrap-kubeconfig created. I have tried with the regular kubeconfig copied to the master node (k3s.yaml) on the bootstrap cluster and this is working.

Finally registered

kubectl get byoh -A
NAMESPACE   NAME             OSNAME   OSIMAGE              ARCH
default     tanzu-master-0   linux    Ubuntu 20.04.5 LTS   amd64
FischerLGLN commented

I had to install iptables on the master and worker nodes too and now my cluster is up and running!

anusha94 commented

I have tried with the regular kubeconfig copied to the master node (k3s.yaml) on the bootstrap cluster and this is working.

Well, this will beat the purpose of having a bootstrap-kubeconfig. The idea is to share a kubeconfig that has restricted access. The regular one probably has admin level privileges.

FischerLGLN commented

@anusha94
Kubeconfig creation way changed with latest k8s versions. I agree, this shouldn't expose admin access, but if one uses a Role with restricted access in this script it will work.

export LOGIN_USER=bootstrapuser
kubectl -n kube-system create serviceaccount $LOGIN_USER

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: $LOGIN_USER
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: "$LOGIN_USER"
EOF

cat << EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: $LOGIN_USER
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: $LOGIN_USER
  namespace: kube-system
EOF

kubectl -n kube-system get secret -o yaml $LOGIN_USER


export USER_TOKEN_NAME=$(kubectl -n kube-system get secret $LOGIN_USER -o=jsonpath='{.metadata.name}')
export USER_TOKEN_VALUE=$(kubectl -n kube-system get secret/${USER_TOKEN_NAME} -o=go-template='{{.data.token}}' | base64 --decode)
export CURRENT_CONTEXT=$(kubectl config current-context)
export CURRENT_CLUSTER=$(kubectl config view --raw -o=go-template='{{range .contexts}}{{if eq .name "'''${CURRENT_CONTEXT}'''"}}{{ index .context "cluster" }}{{end}}{{end}}')
export CLUSTER_CA=$(kubectl config view --raw -o=go-template='{{range .clusters}}{{if eq .name "'''${CURRENT_CLUSTER}'''"}}"{{with index .cluster "certificate-authority-data" }}{{.}}{{end}}"{{ end }}{{ end }}')
export CLUSTER_SERVER=$(kubectl config view --raw -o=go-template='{{range .clusters}}{{if eq .name "'''${CURRENT_CLUSTER}'''"}}{{ .cluster.server }}{{end}}{{ end }}')

cat << EOF > $LOGIN_USER-config
apiVersion: v1
kind: Config
current-context: ${CURRENT_CONTEXT}
contexts:
- name: ${CURRENT_CONTEXT}
  context:
    cluster: ${CURRENT_CONTEXT}
    user: $LOGIN_USER
    namespace: kube-system
clusters:
- name: ${CURRENT_CONTEXT}
  cluster:
    certificate-authority-data: ${CLUSTER_CA}
    server: ${CLUSTER_SERVER}
users:
- name: $LOGIN_USER
  user:
    token: ${USER_TOKEN_VALUE}
EOF
kubectl --kubeconfig $(pwd)/$LOGIN_USER-config get all --all-namespaces
```sh
  • 👍2
ron-arbo commented

same issue here

haiwu commented

hit the same issue with k8s 1.27.2 with --skip-installation flag. The bootstrap user is clusteradmin role. apparently it will have no restrict access.