run byoh-hostagent-linux-amd64 as root -> Unauthorized
knfoo opened this issue · 9 comments
knfoo commented
What steps did you take and what happened:
I am trying to add my VM's to byoh and if I run the byoh-hostagent-linux-amd64 command as a non-root user it works and then kubeadm bootstrap complains that it is an unprivileged user.
However running byoh-hostagent-linux-amd64 as root gives me Unauthorized error
I am running byoh-hostagent-linux-amd64 with --skip-installation as I am running on Debian servers.
What did you expect to happen:
That the node is registered as it is with the non-root user.
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
- Cluster-api-provider-bringyourownhost version: v0.3.1
- Kubernetes version: (use
kubectl version --short): .1.25.3 - OS (e.g. from
/etc/os-release): Debian 11
dharmjit commented
Hi @knfoo, Is it possible to provide the error output you are getting while running the agent with root user?
knfoo commented
@dharmjit
When running as root:
I0127 20:02:41.278012 596590 main.go:230] "msg"="initiated bootstrap kubeconfig flow"
I0127 20:02:41.279589 596590 loader.go:372] Config loaded from file: /home/kn/bootstrap-kubeconfig.conf
I0127 20:02:41.281950 596590 csr.go:120] "msg"="certTimeToExpire" "duration"=31536000000000000
I0127 20:02:41.282601 596590 request.go:1073] Request Body: {"kind":"CertificateSigningRequest","apiVersion":"certificates.k8s.io/v1","metadata":{"name":"byoh-csr-test-k8s01","creationTimestamp":null},"spec":{"request":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlIdk1JR1dBZ0VBTURReEV6QVJCZ05WQkFvVENtSjViMmc2YUc5emRITXhIVEFiQmdOVkJBTVRGR0o1YjJnNgphRzl6ZERwMFpYTjBMV3M0Y3pBeE1Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRVBoODkyRlQ0CkF1b2toQW5zZ25wUENTWW1EdzhQeU5kSXZvSjNhUXdtSXB0eC9pUVpVVnVjZFZyczBWbFdjK1Joc2pMRmc2R3kKaXF0cnlOSHgzUUZpeUtBQU1Bb0dDQ3FHU000OUJBTUNBMGdBTUVVQ0lEdldrL1dkTTFqSUJXQW1YQnUvYW9iZQp1cnJ4ZnBUbGdGT083K0tMWjhkd0FpRUF1TWJKdkd1K3NrZHczcUlPY0dUMUcvSENEV2I1M0VmL3JkVVpibFpLCjJpdz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==","signerName":"kubernetes.io/kube-apiserver-client","expirationSeconds":31536000,"usages":["client auth"]},"status":{}}
I0127 20:02:41.282878 596590 round_trippers.go:466] curl -v -XPOST -H "Authorization: Bearer <masked>" -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: byoh-hostagent-linux-amd64/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://192.168.2.200:6443/apis/certificates.k8s.io/v1/certificatesigningrequests'
I0127 20:02:41.283784 596590 round_trippers.go:510] HTTP Trace: Dial to tcp:192.168.2.200:6443 succeed
I0127 20:02:41.291814 596590 round_trippers.go:553] POST https://192.168.2.200:6443/apis/certificates.k8s.io/v1/certificatesigningrequests 401 Unauthorized in 8 milliseconds
I0127 20:02:41.291865 596590 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 5 ms ServerProcessing 1 ms Duration 8 ms
I0127 20:02:41.291892 596590 round_trippers.go:577] Response Headers:
I0127 20:02:41.291917 596590 round_trippers.go:580] Content-Length: 129
I0127 20:02:41.291944 596590 round_trippers.go:580] Date: Fri, 27 Jan 2023 20:02:41 GMT
I0127 20:02:41.291966 596590 round_trippers.go:580] Audit-Id: cb7b1b84-5de0-4dda-af0e-896a292be80b
I0127 20:02:41.291990 596590 round_trippers.go:580] Cache-Control: no-cache, private
I0127 20:02:41.292011 596590 round_trippers.go:580] Content-Type: application/json
I0127 20:02:41.292148 596590 request.go:1073] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0127 20:02:41.292699 596590 csr.go:129] "msg"="in request certificate" "error"="cannot create certificate signing request: Unauthorized"
E0127 20:02:41.292757 596590 main.go:161] "msg"="bootstrap flow failed" "error"="kubeconfig generation failed: cannot create certificate signing request: Unauthorized"
Running as a non-root user:
kn@test-k8s01:~$ ./byoh-hostagent-linux-amd64 --skip-installation --v 9 --bootstrap-kubeconfig /home/kn/bootstrap-kubeconfig.conf
I0127 20:04:02.300421 596652 loader.go:372] Config loaded from file: /home/kn/.byoh/config
I0127 20:04:02.302069 596652 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: byoh-hostagent-linux-amd64/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://192.168.2.200:6443/api?timeout=32s'
I0127 20:04:02.303200 596652 round_trippers.go:510] HTTP Trace: Dial to tcp:192.168.2.200:6443 succeed
I0127 20:04:02.311673 596652 round_trippers.go:553] GET https://192.168.2.200:6443/api?timeout=32s 200 OK in 9 milliseconds
I0127 20:04:02.311729 596652 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 5 ms ServerProcessing 2 ms Duration 9 ms
I0127 20:04:02.311748 596652 round_trippers.go:577] Response Headers:
I0127 20:04:02.311815 596652 round_trippers.go:580] Audit-Id: 89ec4da5-26bc-4e42-8e11-9c2a212cc96f
I0127 20:04:02.311844 596652 round_trippers.go:580] Cache-Control: no-cache, private
I0127 20:04:02.311865 596652 round_trippers.go:580] Content-Type: application/json
I0127 20:04:02.311891 596652 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 567a3f5e-623f-474c-bf1f-6a734eddfac5
I0127 20:04:02.311920 596652 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: 525684a4-11cd-439b-83de-a9bd066c3f2e
I0127 20:04:02.311937 596652 round_trippers.go:580] Content-Length: 133
I0127 20:04:02.311963 596652 round_trippers.go:580] Date: Fri, 27 Jan 2023 20:04:02 GMT
I0127 20:04:02.312132 596652 request.go:1073] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.18.0.2:6443"}]}
I0127 20:04:02.312521 596652 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: byoh-hostagent-linux-amd64/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://192.168.2.200:6443/apis?timeout=32s'
I0127 20:04:02.314854 596652 round_trippers.go:553] GET https://192.168.2.200:6443/apis?timeout=32s 200 OK in 2 milliseconds
I0127 20:04:02.314888 596652 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 1 ms Duration 2 ms
I0127 20:04:02.314900 596652 round_trippers.go:577] Response Headers:
I0127 20:04:02.314924 596652 round_trippers.go:580] Cache-Control: no-cache, private
I0127 20:04:02.314937 596652 round_trippers.go:580] Content-Type: application/json
I0127 20:04:02.314962 596652 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 567a3f5e-623f-474c-bf1f-6a734eddfac5
I0127 20:04:02.314973 596652 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: 525684a4-11cd-439b-83de-a9bd066c3f2e
I0127 20:04:02.315002 596652 round_trippers.go:580] Date: Fri, 27 Jan 2023 20:04:02 GMT
I0127 20:04:02.315013 596652 round_trippers.go:580] Audit-Id: 35c376ea-f2de-44e9-b349-42efa18fb22c
I0127 20:04:02.315209 596652 request.go:1073] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1","version":"v1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1","version":"v1"}},{"name":"authorization.k8s.io","versions":[{"groupVersion":"authorization.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"authorization.k8s.io/v1","version":"v1"}},{"name":"autoscaling","versions":[{"groupVersion":"autoscaling/v2","version":"v2"},{"groupVersion":"autoscaling/v1","version":"v1"},{"groupVersion":"autoscaling/v2beta2","version":"v2beta2"}],"preferredVersion":{"groupVersion":"autoscaling/v2","version":"v2"}},{"name":"batch","versions":[{"groupVersion":"batch/v1","version":"v1"}],"preferredVersion":{"groupVersion":"batch/v1","version":"v1"}},{"name":"certificates.k8s.io","versions":[{"groupVersion":"certificates.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"certificates.k8s.io/v1","version":"v1"}},{"name":"networking.k8s.io","versions":[{"groupVersion":"networking.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"networking.k8s.io/v1","version":"v1"}},{"name":"policy","versions":[{"groupVersion":"policy/v1","version":"v1"}],"preferredVersion":{"groupVersion":"policy/v1","version":"v1"}},{"name":"rbac.authorization.k8s.io","versions":[{"groupVersion":"rbac.authorization.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"rbac.authorization.k8s.io/v1","version":"v1"}},{"name":"storage.k8s.io","versions":[{"groupVersion":"storage.k8s.io/v1","version":"v1"},{"groupVersion":"storage.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"storage.k8s.io/v1","version":"v1"}},{"name":"admissionregistration.k8s.io","versions":[{"groupVersion":"admissionregistration.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"admissionregistration.k8s.io/v1","version":"v1"}},{"name":"apiextensions.k8s.io","versions":[{"groupVersion":"apiextensions.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apiextensions.k8s.io/v1","version":"v1"}},{"name":"scheduling.k8s.io","versions":[{"groupVersion":"scheduling.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"scheduling.k8s.io/v1","version":"v1"}},{"name":"coordination.k8s.io","versions":[{"groupVersion":"coordination.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"coordination.k8s.io/v1","version":"v1"}},{"name":"node.k8s.io","versions":[{"groupVersion":"node.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"node.k8s.io/v1","version":"v1"}},{"name":"discovery.k8s.io","versions":[{"groupVersion":"discovery.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"discovery.k8s.io/v1","version":"v1"}},{"name":"flowcontrol.apiserver.k8s.io","versions":[{"groupVersion":"flowcontrol.apiserver.k8s.io/v1beta2","version":"v1beta2"},{"groupVersion":"flowcontrol.apiserver.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"flowcontrol.apiserver.k8s.io/v1beta2","version":"v1beta2"}},{"name":"acme.cert-manager.io","versions":[{"groupVersion":"acme.cert-manager.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"acme.cert-manager.io/v1","version":"v1"}},{"name":"cert-manager.io","versions":[{"groupVersion":"cert-manager.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"cert-manager.io/v1","version":"v1"}},{"name":"ipam.cluster.x-k8s.io","versions":[{"groupVersion":"ipam.cluster.x-k8s.io/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"ipam.cluster.x-k8s.io/v1alpha1","version":"v1alpha1"}},{"name":"runtime.cluster.x-k8s.io","versions":[{"groupVersion":"runtime.cluster.x-k8s.io/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"runtime.cluster.x-k8s.io/v1alpha1","version":"v1alpha1"}},{"name":"addons.cluster.x-k8s.io","versions":[{"groupVersion":"addons.cluster.x-k8s.io/v1beta1","version":"v1beta1"},{"groupVersion":"addons.cluster.x-k8s.io/v1alpha4","version":"v1alpha4"},{"groupVersion":"addons.cluster.x-k8s.io/v1alpha3","version":"v1alpha3"}],"preferredVersion":{"groupVersion":"addons.cluster.x-k8s.io/v1beta1","version":"v1beta1"}},{"name":"bootstrap.cluster.x-k8s.io","versions":[{"groupVersion":"bootstrap.cluster.x-k8s.io/v1beta1","version":"v1beta1"},{"groupVersion":"bootstrap.cluster.x-k8s.io/v1alpha4","version":"v1alpha4"},{"groupVersion":"bootstrap.cluster.x-k8s.io/v1alpha3","version":"v1alpha3"}],"preferredVersion":{"groupVersion":"bootstrap.cluster.x-k8s.io/v1beta1","version":"v1beta1"}},{"name":"cluster.x-k8s.io","versions":[{"groupVersion":"cluster.x-k8s.io/v1beta1","version":"v1beta1"},{"groupVersion":"cluster.x-k8s.io/v1alpha4","version":"v1alpha4"},{"groupVersion":"cluster.x-k8s.io/v1alpha3","version":"v1alpha3"}],"preferredVersion":{"groupVersion":"cluster.x-k8s.io/v1beta1","version":"v1beta1"}},{"name":"clusterctl.cluster.x-k8s.io","versions":[{"groupVersion":"clusterctl.cluster.x-k8s.io/v1alpha3","version":"v1alpha3"}],"preferredVersion":{"groupVersion":"clusterctl.cluster.x-k8s.io/v1alpha3","version":"v1alpha3"}},{"name":"controlplane.cluster.x-k8s.io","versions":[{"groupVersion":"controlplane.cluster.x-k8s.io/v1beta1","version":"v1beta1"},{"groupVersion":"controlplane.cluster.x-k8s.io/v1alpha4","version":"v1alpha4"},{"groupVersion":"controlplane.cluster.x-k8s.io/v1alpha3","version":"v1alpha3"}],"preferredVersion":{"groupVersion":"controlplane.cluster.x-k8s.io/v1beta1","version":"v1beta1"}},{"name":"infrastructure.cluster.x-k8s.io","versions":[{"groupVersion":"infrastructure.cluster.x-k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"infrastructure.cluster.x-k8s.io/v1beta1","version":"v1beta1"}}]}
I0127 20:04:02.316195 596652 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: byoh-hostagent-linux-amd64/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://192.168.2.200:6443/apis/authorization.k8s.io/v1?timeout=32s'
I0127 20:04:02.316271 596652 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: byoh-hostagent-linux-amd64/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://192.168.2.200:6443/apis/infrastructure.cluster.x-k8s.io/v1beta1?timeout=32s'
I0127 20:04:02.316489 596652 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: byoh-hostagent-linux-amd64/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://192.168.2.200:6443/apis/autoscaling/v2?timeout=32s'
I0127 20:04:02.316620 596652 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: byoh-hostagent-linux-amd64/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://192.168.2.200:6443/api/v1?timeout=32s'
I0127 20:04:02.316756 596652 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: byoh-hostagent-linux-amd64/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://192.168.2.200:6443/apis/autoscaling/v1?timeout=32s'
I0127 20:04:02.316776 596652 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: byoh-hostagent-linux-amd64/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://192.168.2.200:6443/apis/events.k8s.io/v1?timeout=32s'
As you can see I use the same bootstrap config for both root and none-root user and with the non-root user it works.
dharmjit commented
"msg"="in request certificate" "error"="cannot create certificate signing request: Unauthorized"
This error occurs when the bootstrap kubeconfig doesn't have the right permissions. I guess you have followed the steps documented here. The RBAC is defined in ClusterRoleBinding named byoh-csr-creator-clusterrole-binding. Could you check if this ClusterRoleBinding exists in your management cluster?
with the non-root user it works.
In the logs, you could see that it already finds kubeconfig in the /home/kn/.byoh/config. In this case, it skips the bootstrap kubeconfig flow.
knfoo commented
@dharmjit thank you for pointing out what I missed in the logs I copied the config from /home/kn to /root and now it works :)
I ran into the next problem that maybe you can help with ?
when installing the cluster it never moves past the controlplane node...
kubectl get secret/byoh-cluster-kubeconfig -o json | jq -r .data.value | base64 --decode > kubeconfig.yaml
kn@test-k8s00:~$ KUBECONFIG=kubeconfig.yaml k get nodes
NAME STATUS ROLES AGE VERSION
test-k8s01 NotReady control-plane 4m10s v1.24.2
from the controler log:
I0202 05:14:57.684440 1 byomachine_controller.go:89] controller/byomachine "msg"="Reconcile request received" "name"="byoh-cluster-control-plane-c4k4z" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"
I0202 05:14:57.684860 1 byomachine_controller.go:191] controller/byomachine "msg"="Fetching an attached ByoHost" "name"="byoh-cluster-control-plane-c4k4z" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"
I0202 05:14:57.685456 1 byomachine_controller.go:208] controller/byomachine "msg"="Successfully fetched an attached Byohost" "byohost"="test-k8s01" "name"="byoh-cluster-control-plane-c4k4z" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"
I0202 05:14:57.685999 1 byomachine_controller.go:236] controller/byomachine "msg"="Reconciling ByoMachine" "cluster"="byoh-cluster" "name"="byoh-cluster-control-plane-c4k4z" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"
I0202 05:14:57.698403 1 byomachine_controller.go:455] controller/byomachine "msg"="Installer config is not ready, requeuing" "cluster"="byoh-cluster" "name"="byoh-cluster-control-plane-c4k4z" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"
I0202 05:15:07.699967 1 byomachine_controller.go:89] controller/byomachine "msg"="Reconcile request received" "name"="byoh-cluster-control-plane-c4k4z" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"
I0202 05:15:07.700374 1 byomachine_controller.go:191] controller/byomachine "msg"="Fetching an attached ByoHost" "name"="byoh-cluster-control-plane-c4k4z" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"
I0202 05:15:07.700542 1 byomachine_controller.go:208] controller/byomachine "msg"="Successfully fetched an attached Byohost" "byohost"="test-k8s01" "name"="byoh-cluster-control-plane-c4k4z" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"
I0202 05:15:07.700872 1 byomachine_controller.go:236] controller/byomachine "msg"="Reconciling ByoMachine" "cluster"="byoh-cluster" "name"="byoh-cluster-control-plane-c4k4z" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"
I0202 05:15:07.716283 1 byomachine_controller.go:455] controller/byomachine "msg"="Installer config is not ready, requeuing" "cluster"="byoh-cluster" "name"="byoh-cluster-control-plane-c4k4z" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"
dharmjit commented
Hi @knfoo, Have you figured this out? I guess you are creating 1 CP 1 worker node cluster. Can you share the agent logs for the host which is supposed to become the worker node?
knfoo commented
@dharmjit thanks to that hint I found that there is a preflight error.
Feb 07 19:50:19 test-k8s03 byoh-hostagent-linux-amd64[1555859]: [WARNING SystemVerification]: missing optional cgroups: blkio
Feb 07 19:50:19 test-k8s03 byoh-hostagent-linux-amd64[1555859]: error execution phase preflight: [preflight] Some fatal errors occurred:
Feb 07 19:50:19 test-k8s03 byoh-hostagent-linux-amd64[1555859]: [ERROR SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.10.0-20-amd64\n", err: exit status 1
Feb 07 19:50:19 test-k8s03 byoh-hostagent-linux-amd64[1555859]: [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
I tried to fix that by:
https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta2/ more specifically `ignorePreflightErrors` but but cluster-api versions installed with byoh is v1beta1
so I am kind of stuck on this new error
knfoo commented
I got past that error and now I get by installing the linux-image package in my Xen VM's.
dharmjit commented
Kubeadm preflight checks are related to system/kernel configurations. I guess you are past that. Are you facing any other issues?
knfoo commented
Yes I now have a running cluster 💯 thank you for all your help 👍