Pre Install Checker Reports Namespace Admin Permissions - Insufficient.
lorenzk1213 opened this issue · 20 comments
Hi,
We have used viya4-iac to deploy the AWS environment. and used viya4-deployment DAC to deploy the baseline components,
Right now, tried to run the viy4-ark pre-install checker,
My Viya-Ark Pre-Install Checker Reports indequate permission for the Namespace Admin,
Can pls help on how to address this issue?
Thanks,
@lorenzk1213 these are permissions set up by whoever or whatever setup your kubernetes server. If you used IAC to create it then you may want to bring this up to them. If your kubernetes was created by someone else or your IT department then you''ll need to work with them to get your permissions corrected.
@lorenzk1213
Please turn on the debug option and forward the logs to us. I'll take a look.
Please find attached log with DEBUG enabled.
viya_pre_install_log_2022-05-26T06_27_46.log
*For security purposes, have deleted top portion of the log which contains AWS information
@lorenzk1213 Thank you for the debug log.
- Please issue the following command from your kubectl client machine and share the output if possible.
kubectl -n default api-resources -o wide
I think you have permissions restrictions. - Forward a copy of the report.
- Capture the command line options specified for the tool.
- The following failures are unexpected:
Though these resources were created successfully, the tool is not able to find them when it tries to delete them. Looking into it.
Thank you
@lasiva
I see the following 3 having Delete roles from the api-resources output. Could this be possibly a reporting tool bug?
roles rbac.authorization.k8s.io/v1 true Role [create delete deletecollection get list patch update watch]
rolebindings rbac.authorization.k8s.io/v1 true RoleBinding [create delete deletecollection get list patch update watch]
serviceaccounts sa v1 true ServiceAccount [create delete deletecollection get list patch update watch]
The permissions seem to be available, but I see some issues .
- The end of the output from api-resources in your attached file shows the following:
error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request.
This results in the tool capturing the following:
Command 'kubectl -n default api-resources -o wide' returned non-zero exit status 1,
The tool is interpreting that as insufficient Permissions.
You might check with following command to make sure there are no issues on the cluster.
kubectl get pods -n kube-system
Please Check if some pod are down.
- Also the failed deletes are unable to find the resources to delete. Like following.
"/home/925408/viya4-ark-1.8.0/pre_install_report/library/utils/viya-rolebinding.yaml": rolebindings.rbac.authorization.k8s.io "viyasrolebinding" not found\n' error_out b''
The tools is interpreting Not Found as Insufficient permissions which is misleading.
I'm not sure at this point why the tool is unable to find resources that it successfully created. Have not hit that before.
- Please provide the output of the "kubectl version" command also.
@lorenzk1213
I'm attempting to reproduce your error. I'll run the tool against default namespace in a clean cluster.
In the mean time we think you could go ahead and try a full deployment and not let the pre-check tool block you.
fyi... we are closed for national holiday this Monday 30th. I may not be able to get back till late Monday or Tuesday,
Thank you
@lorenzk1213
I was able to recreate your error scenario.
- Created an AWS cluster with no Viya deployment
- Deleted the metrics server pod
- Issued the kubectl api-services -n kube-system command and it resulted in the same error: "error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
- Finally ran the pre_install_check tool and got the same results as the one supplied by you. The pre_install_check tool will fail some checks if the api-resources commands returns non-zero return code.
- Is there a metrics server installed and running? Let us know.
@lasiva Yes we do have metric server installed and running
@lorenzk1213
The kubectl api-services -n kube-system command resulted in an error on you system: "error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request". The error was in your attachment attachment per my earlier comment.
Please resolve that issue and try again. Though the metrics server is running, the command results in non-zero return code when you run the tool.
What helm chart is being used for the metrics servers? With k8s 1.19 and above it should be 5.11 and above : https://artifacthub.io/packages/helm/bitnami/metrics-server/5.11.0 This supports the metrics-server 0.6.0 and higher which is targeted to 1.19+
Could this be related to the metrics-server issue in sassoftware/viya4-deployment#230 ?
@thpang thanks for following up. We will look at the tool dependency on the metrics server.
We dont have a dependency on the metrics server but tool is looking for a clean rc from the following command:
kubectl -n default api-resources -o wide (which is failing on this issue due to metrics server).
Will investigate options further.
If kubectl returns a non-zero code we expect it means failure. I'd recommend entering an issue against kubectl if there's not already one entered for it.
I stand corrected, it turns out the use of this specific kubectl command as it relates to Viya4-ark is only to determine its return code in one particular if conditional, but not actually using the data returned by the command. @lasiva has determined that this conditional is not necessary for the intended purpose of the function it is in and can be safely removed without affecting the intended behavior.
This issue is addressed in Release 1.8.1.