dynu with cert-manager 1.9.1 on arm: RBACs problem
rbaumgar opened this issue · 9 comments
I have installed cert-manager 1.9.1
dynu latest version on arm.
dynu-webhook log
I0829 19:50:00.984703 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0829 19:50:00.984990 1 shared_informer.go:255] Waiting for caches to sync for RequestHeaderAuthRequestController
I0829 19:50:00.984761 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0829 19:50:00.985142 1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0829 19:50:00.984774 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0829 19:50:00.986033 1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0829 19:50:00.986042 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tls/tls.crt::/tls/tls.key"
I0829 19:50:00.985992 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0829 19:50:00.985932 1 secure_serving.go:210] Serving securely on [::]:10250
I0829 19:50:00.987461 1 apf_controller.go:317] Starting API Priority and Fairness config controller
W0829 19:50:00.997963 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-dynu-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0829 19:50:00.998303 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1beta2.FlowSchema: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-dynu-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
W0829 19:50:01.004129 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-dynu-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0829 19:50:01.004464 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1beta2.PriorityLevelConfiguration: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-dynu-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
I0829 19:50:01.086246 1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0829 19:50:01.086270 1 shared_informer.go:262] Caches are synced for RequestHeaderAuthRequestController
I0829 19:50:01.086342 1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
W0829 19:50:02.154477 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-dynu-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0829 19:50:02.154663 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1beta2.PriorityLevelConfiguration: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-dynu-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
SA system:serviceaccount:cert-manager:cert-manager-dynu-webhook has not sufficient rights for flowcontrol.apiserver.k8s.io
found similar issue puzzle/cert-manager-webhook-dnsimple#8
I believe I saw these messages in my installation too. I shall be back home to confirm that in a couple of days. However, they seem inconsequential as my system was able to get the certificate signed successfully and I did not have time to look into them further. Did it work for you otherwise?
I was able to fix the RBAC problem by applying the following clusterrole and clusterrolebinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
meta.helm.sh/release-name: cert-manager-dynu-webhook
meta.helm.sh/release-namespace: cert-manager
labels:
app: dynu-webhook
app.kubernetes.io/managed-by: Helm
chart: dynu-webhook-0.1.1
heritage: Helm
release: cert-manager-dynu-webhook
name: cert-manager-dynu-webhook:flowcontrol-solver
rules:
- apiGroups:
- "flowcontrol.apiserver.k8s.io"
resources:
- 'prioritylevelconfigurations'
- 'flowschemas'
verbs:
- 'list'
- 'watch'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
meta.helm.sh/release-name: cert-manager-dynu-webhook
meta.helm.sh/release-namespace: cert-manager
labels:
app: dynu-webhook
app.kubernetes.io/managed-by: Helm
chart: dynu-webhook-0.1.1
heritage: Helm
release: cert-manager-dynu-webhook
name: cert-manager-dynu-webhook:flowcontrol-solver
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cert-manager-dynu-webhook:flowcontrol-solver
subjects:
- kind: ServiceAccount
name: cert-manager-dynu-webhook
namespace: cert-manager
now I have the following error messages
W0829 21:24:32.439246 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1beta2.PriorityLevelConfiguration: the server could not find the requested resource
E0829 21:24:32.439742 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1beta2.PriorityLevelConfiguration: failed to list *v1beta2.PriorityLevelConfiguration: the server could not find the requested resource
W0829 21:25:00.737869 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1beta2.FlowSchema: the server could not find the requested resource
E0829 21:25:00.737959 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1beta2.FlowSchema: failed to list *v1beta2.FlowSchema: the server could not find the requested resource
I found now that I am using dynu version 0.1.1 and 0.1.2. Installed 2 hours ago.
Any idea why?
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
cert-manager-dynu-webhook cert-manager 1 2022-08-29 19:49:30.930351429 +0000 UTC deployed dynu-webhook-0.1.1 1.0
The only difference between these two versions is the readme file. I cannot comment on the new errors you see now.
both objects exist, but I don't know which name the webhook is looking for and who should create it...
# kubectl get prioritylevelconfiguration
NAME TYPE ASSUREDCONCURRENCYSHARES QUEUES HANDSIZE QUEUELENGTHLIMIT AGE
catch-all Limited 5 <none> <none> <none> 2d14h
exempt Exempt <none> <none> <none> <none> 2d14h
global-default Limited 20 128 6 50 2d14h
leader-election Limited 10 16 4 50 2d14h
system Limited 30 64 6 50 2d14h
workload-high Limited 40 128 6 50 2d14h
workload-low Limited 100 128 6 50 2d14h
[root@microshift ~]# kubectl get flowschema
NAME PRIORITYLEVEL MATCHINGPRECEDENCE DISTINGUISHERMETHOD AGE MISSINGPL
exempt exempt 1 <none> 2d14h False
system-leader-election leader-election 100 ByUser 2d14h False
workload-leader-election leader-election 200 ByUser 2d14h False
system-nodes system 500 ByUser 2d14h False
kube-controller-manager workload-high 800 ByNamespace 2d14h False
kube-scheduler workload-high 800 ByNamespace 2d14h False
kube-system-service-accounts workload-high 900 ByNamespace 2d14h False
service-accounts workload-low 9000 ByUser 2d14h False
global-default global-default 9900 ByUser 2d14h False
catch-all catch-all 10000 ByUser 2d14h False
very interesting! Audit log shows following error
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"df17037b-902f-4779-8dae-d49b4e8699c0","stage":"ResponseComplete","requestURI":"/apis/flowcontrol.apiserver.k8s.io/v1beta2/flowschemas?limit=500\u0026resourceVersion=0","verb":"list","user":{"username":"system:serviceaccount:cert-manager:cert-manager-dynu-webhook","uid":"890b445d-a09a-4037-a443-65081814a78b","groups":["system:serviceaccounts","system:serviceaccounts:cert-manager","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["cert-manager-dynu-webhook-79f445dfd4-dr285"],"authentication.kubernetes.io/pod-uid":["751180b3-ba97-4f8d-abfb-dc81b103791b"]}},"sourceIPs":["10.42.0.10"],"userAgent":"webhook/v0.0.0 (linux/arm64) kubernetes/$Format","objectRef":{"resource":"flowschemas","apiGroup":"flowcontrol.apiserver.k8s.io","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"requestReceivedTimestamp":"2022-08-30T13:12:58.533325Z","stageTimestamp":"2022-08-30T13:12:58.577099Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding "cert-manager-dynu-webhook:flowcontrol-solver" of ClusterRole "cert-manager-dynu-webhook:flowcontrol-solver" to ServiceAccount "cert-manager-dynu-webhook/cert-manager""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a2332bde-38d9-4fca-8beb-58a9accb70be","stage":"ResponseComplete","requestURI":"/apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations?limit=500\u0026resourceVersion=0","verb":"list","user":{"username":"system:serviceaccount:cert-manager:cert-manager-dynu-webhook","uid":"890b445d-a09a-4037-a443-65081814a78b","groups":["system:serviceaccounts","system:serviceaccounts:cert-manager","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["cert-manager-dynu-webhook-79f445dfd4-dr285"],"authentication.kubernetes.io/pod-uid":["751180b3-ba97-4f8d-abfb-dc81b103791b"]}},"sourceIPs":["10.42.0.10"],"userAgent":"webhook/v0.0.0 (linux/arm64) kubernetes/$Format","objectRef":{"resource":"prioritylevelconfigurations","apiGroup":"flowcontrol.apiserver.k8s.io","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"requestReceivedTimestamp":"2022-08-30T13:13:50.036863Z","stageTimestamp":"2022-08-30T13:13:50.145252Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding "cert-manager-dynu-webhook:flowcontrol-solver" of ClusterRole "cert-manager-dynu-webhook:flowcontrol-solver" to ServiceAccount "cert-manager-dynu-webhook/cert-manager""}}
oh I found the reason for the errors.
I have only version v1beta1 of flowcontrol, but the pod is looking for version v1beta2
# kubectl api-resources --api-group=flowcontrol.apiserver.k8s.io
NAME SHORTNAMES APIVERSION NAMESPACED KIND
flowschemas flowcontrol.apiserver.k8s.io/v1beta1 false FlowSchema
prioritylevelconfigurations flowcontrol.apiserver.k8s.io/v1beta1 false PriorityLevelConfiguration
This has been fixed in latest update so can be closed