New LIST/WATCH permission requirement for aggregated apiservers & versioning aggregated apiservers
munnerz opened this issue · 6 comments
What happened:
In cert-manager, we provide a library github.com/jetstack/cert-manager/pkg/acme/webhook that wraps k8s.io/apiserver to allow users to create their own webhooks that integrate with parts of our own project. This is to facilitate trusted communication with our extension points (webhooks) via the apiserver's aggregated API trust mechanism.
Upon upgrading our client library dependencies for k8s.io/* to 1.17.0 from 1.16.3, our tests began failing with: E1216 13:01:10.267268 1 reflector.go:156] external/io_k8s_client_go/tools/cache/reflector.go:108: Failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:e2e-tests-acme-dns01-sample-webhook-8v7fv:wh-cm-e2e-acme-dns01-sample-webhook-example-webhook" cannot list resource "configmaps" in API group "" in the namespace "kube-system".
RBAC permissions for the webhooks are granted via a RoleBinding between the webhook's service account and the role extension-apiserver-authentication-reader being created in the kube-system namespace:
Upon upgrading our client libraries, the new version of k8s.io/apiserver attempted to utilise the list verb in order to read the CA data.
In versions of Kubernetes prior to v1.17, the GET alone was used to read the CA ConfigMap. Because our end-to-end tests are currently configured to run against Kubernetes 1.15, these tests began failing as the aforementioned Role that is created/managed by Kubernetes itself does not contain the list and watch verbs.
What you expected to happen:
This highlights another versioning paradigm to be aware of & documented: aggregated apiservers must have their client libraries updated after upgrading the Kubernetes installation. I'm uncertain if there's any other version skew considerations to take account of, but all the same it should be documented in the version skew policy: https://kubernetes.io/docs/setup/release/version-skew-policy/#supported-component-upgrade-order
How to reproduce it (as minimally and precisely as possible):
- Create a Kubernetes cluster running version 1.16 or below
- Deploy an aggregated apiserver built using Kubernetes 1.17
k8s.io/apiserver - Grant RBAC permissions for this aggregated apiserver by binding to the system-provisioned
extension-apiserver-authentication-readerRole
Anything else we need to know?:
There probably isn't any actionable code change here, unless this particular instance of the issue is deemed to be severe enough to warrant it.
/cc @liggitt @deads2k
/sig auth
/sig api-machinery
/kind documentation
I'm not sure if this should really be considered a bug 🐛
/remove-kind bug
Yeah, it's a bug. We fixed the permission in 1.17.0 and should backport.
Workaround is to create your own role or modify the stock role
/assign @deads2k
backports are merged
/close
@liggitt: Closing this issue.
In response to this:
backports are merged
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.