KCP API resource object integration with managed-gitops
Closed this issue · 3 comments
APIExport and APIResourceSchema KCP API for the GITOPS SERVICE in Red Hat AppStudio
Before deep diving into the issue, let's understand briefly what APIExport, APIResourceSchema, and APIBindings stand for, and how they are interrelated to each other.
(Note: This information as of writing Aug 10, is taken from the kcp-dev/kcp codebase, and discussion from the kcp team folks, please correct me if I'm wrong in the below comment in the same issue)
APIResourceSchema:
APIResourceSchema describes a resource, identified by (group, version, resource, schema). An APIResourceSchema is immutable and cannot be deleted if they are referenced by an APIExport in the same workspace.
APIExport:
APIExport registers an API and implementation to allow consumption by others through APIBindings. Different APIExport in a workspace can share a common identity, or have different ones.
latestResourceSchemas (that is used in #194) records the latest APIResourceSchemas that are exposed with this APIExport.
permissionClaims (that is used in #194) make resources available in APIExport's virtual workspace that is not part of the actual APIExport resources.
APIBindings:
APIBindings enables a set of resources and their behavior through an external service provider in this workspace. The service provider uses an APIExport to expose the API. There are two phases for the APIBinding "Binding and Bound",
As of filing this story, we will not be able to claim APIBindings for write-access but there is a PR open on kcp-dev/kcp #1563 which talks about having read access to APIBindings.
Inter-relation b/w these three resources above:
Q: So to map out from the above terminologies, when we register an API using APIExport against APIResourceSchema with the help of APIBindings, the APIExport creates a virtual workspace that the end user is unaware of (it is different from the logical workspace in which we are in. So for such scenarios, we might need permissionclaim labels so those resources (secrets/cm) to go to workspaces, or did I get this all wrong?
A: the virtual workspace apiserver is your (highly controlled, security-wise) access point to tenant workspaces. APIBindings normally give you as APIExport owner access to only the resources exported by you. The permission claims allow you to ask the user for more access, to other resources like secrets and namespaces. If you do so and the user agrees to your claim requests, then you will see those other resources in the virtual workspace apiserver.
(Thanks to Stefan for this explanation)
What's the issue at the moment and what's the issue we should keep in mind for the future?
At this moment with the PR #188 @chetan-rns is working on, testing against the #194 PR we found at that in order to make the gitops-service controller happy, we need to include APIExport's permissionclaim (in which in order controller to work we needed "secrets" and "namespaces") which is referenced in the PR opened.
But, as talking to Stefan about the same concern today there was a concern raised, and I'm copying paste the same to maintain transparency:
my gut feeling is that for Gitops we need something more than the current claims. You as Gitops provider need access to the resources that the user wants to reconcile via ArgoCD (or whatever the tool is). You as Gitops provider don't know which resources in advance. So basically you will have to claim everything just to be on the safe side.
"claiming everything" is not possible today we want to discourage to "claim everything" for service providers in general.So this feels like we will depend a lot on the user UI oder CLI to make this both convenient, easy and yet as secure as possible.
References:
PRs/Stories?
- JIRA Story: GITOPSRVC-188.
- PR by @samyak-jn: #194.
- PR by @chetan-rns : #188.
FYI For the management side of the people, I agree we use JIRA for issues, but for both the team communication this is the preferred choice, but to track in JIRA, I have created a JIRA story for the same, GITOPSRVC-205.
@samyak-jn thanks for writing this up, I agree with everything you mentioned above.
Here is my current solution to the issues described in #198 and GITOPSRVCE-205:
The Core GitOps Service itself (specifically the backend component) will only need access to the following resources in a user's KCP workspace:
- GitopsDeployment
- GitOpsDeploymentRepositoryCredentials
- GitOpsDeploymentManagedEnvironment
- GitOpsDeploymentSyncRun
- Secret (the above resources reference Secrets, for example, repository credentials are contained in a Secret)
- Namespace
This list of resources will allow the GitOps Service to see API resources that are meant for it to handle (GitOpsDeployment*), plus the K8s resources that are referenced by those API resources (Secrets, Namespace). (I'm ignoring appstudio-controller/appstudio-shared for the purposes of this comment)
So adding Secret and Namespace to permission claims of the APIExport handles this.
However, Argo CD also needs to be able to deploy to user (logical) workspaces. And, as you noted, Argo CD will need to deploy any resource.
Short Term Solution
So, my current solution -- using what is currently implemented in GitOps Service -- is as follows:
TLDR: For each KCP user workspace that Argo CD will need to deploy to, we need to ensure the following is created:
- A new ServiceAccount: this account is what Argo CD will use to deploy
- Or, we could just use 'default' ServiceAccount, in some namespace, for now.
- New Role/RoleBinding/ClusterRole/ClusterRoleBindings: Grants the ServiceAccount the ability to read/write any resource in the workspace
- ServiceAccount's Secret (Secret of type
kubernetes.io/service-account-token): Contains the login credentials for the ServiceAccount, that allow Argo CD to 'login' to the KCP workspace as the service account
How this works:
- Argo CD needs a Kubernetes API URL and Login credentials in order to access the workspace
- the KCP workspace URL, and the credentials from the Secret, will work well for this.
- Argo CD needs to be able to read/write any resource in the workspace
- the (cluster)role/rolebindings will allow this
So, step by step, this is what is required in order to allow Argo CD to deploy to a workspace:
- (A user will need to do all of the following steps themselves; we can probably script it initially)
- Create a ServiceAccount in the user's KCP workspace
- Create a
service-account-tokenSecret targeting that ServiceAccount
- type of 'kubernetes.io/service-account-token'
- annotation of: "kubernetes.io/service-account.name": (name of service account from step 1)
- Convert the service account token (from the Secret) and API URL into a kubeconfig, and store that in a Secret
- Create GitOpsDeploymentManagedEnvironment resource in the user workspace, pointing to the secret from the previous step
- Create GitOpsDeployment resource in user workspace, pointing to the managed environment
How this works:
- GitOps Service will have access to the GitOpsDeploymentManagedEnvironment, and the Secret it points to, via the virtual workspace
- The Secret contains login credentials for the KCP workspace
- The ManagedEnvironment references the Secret
- Both are accessible via the permission claims, as above.
- The GitOps Service reads the ManagedEnvironment, and Secret, and creates a corresponding Argo CD cluster secret in Argo CD
- This allows Argo CD to deploy to the user workspace
- This logic takes advantage of the GitOpsDeploymentManagedEnvironment feature, which already exists in our codebase, as of M6.
This solution will work, but has major disadvantages: it requires the user to go through a bunch of configuration steps (creating the ManagedEnvironment, Secret, Roles/RoleBindings, etc) in order for the GitOps Service (and Argo CD) to deploy to their workspace.
However, a better solution exists.
Short/Medium Term Solution
In the short/medium term, it will look like this:
- Whenever a new user workspace is detected, the workspace initializer (a k8s controller that AFAIK we will own) creates the required resources in the workspace
- The workspace initializer resource creation code will run whenever a new workspace is created
- We can introduce our own code here, to create any objects that we need to create.
- In this case, we will create ServiceAccount/(Cluster)Role/RoleBindings/Secret automatically, for the user. (Using the same process as above)
- This ensures that the resources exist for Argo CD to deploy to the Namespace, and that it has the necessary permissions to do so.
So using the 'workspace initializer' mechanism, we will ensure that the Workspace is properly initialized with ServiceAccount/(Cluster)Role/RoleBinding/Secret, and that GitOps Service/Argo CD is configured to use this data, so that it can deploy directly to the KCP workspace.
The proposed short/medium term solution seems reasonable to me.