Crossplane Kubernetes Provider does not create the status fields in target k8s cluster
ashwinshirva opened this issue · 8 comments
What happened?
I am using crossplane claim, XR, composition and kubernetes provider to create a kubernetes custon resource in an target kubernetes cluster. The custom resource is getting created in the target cluster, but only with the spec fields. I have included the status field as well in the managed resource defined in crossplane composition, but looks the kubernetes provider does not create the status part, it creates only the spec part of the custom resource in external cluster
How can we reproduce it?
Claim YAML
apiVersion: example.org/v1alpha1
kind: MyCr
metadata:
namespace: default
name: my-cr-1
spec:
parameters:
field2: "sample field2"
compositionRef:
name: mycrcomposition
writeConnectionSecretToRef:
name: mymy-cr-1-details
XRD YAML:
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
name: xmycrs.example.org
spec:
group: example.org
names:
kind: XMyCr
plural: xmycrs
claimNames:
kind: MyCr
plural: mycrs
connectionSecretKeys:
- hostname
defaultCompositionRef:
name: mycrcomposition
versions:
- name: v1alpha1
served: true
referenceable: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
parameters:
type: object
properties:
field2:
type: string
required:
- field2
required:
- parameters
status:
type: object
properties:
phase:
description: Phase that the CR is in currently.
type: string
Composition YAML
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
name: mycrcomposition
labels:
crossplane.io/xrd: xmycr.example.org
crtype: myresource
spec:
compositeTypeRef:
apiVersion: example.org/v1alpha1
kind: XMyCr
writeConnectionSecretsToNamespace: crossplane-system
resources:
- name: myresource
base:
apiVersion: kubernetes.crossplane.io/v1alpha1
kind: Object
metadata:
name: myresource-sample-3
namespace: "default"
spec:
forProvider:
manifest:
apiVersion: example.com/v1
kind: MyResource
metadata:
name: myresource-sample-3
namespace: default
spec:
field1: "value1"
field2: "value2"
status:
message: "Sample status message"
phase: "Running"
providerConfigRef:
name: kubernetes-provider
patches:
- type: FromCompositeFieldPath
fromFieldPath: spec.parameters.field2
toFieldPath: spec.forProvider.manifest.spec.field2
policy:
fromFieldPath: Required
readinessChecks:
- type: None
Custom Resource Definition (CRD):
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: myresources.example.com
spec:
group: example.com
names:
kind: MyResource
singular: myresource
plural: myresources
shortNames:
- mr
scope: Namespaced
versions:
- name: v1
served: true
storage: true
subresources:
status: {}
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
field1:
type: string
field2:
type: string
required:
- field1
- field2
status:
type: object
properties:
message:
type: string
phase:
type: string
required:
- message
- phase
Custom resource created by kubenetes provider in target cluster:
apiVersion: example.com/v1
kind: MyResource
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"example.com/v1","kind":"MyResource","metadata":{"name":"myresource-sample-3","namespace":"default"},"spec":{"field1":"value1","field2":"sample
field2"},"status":{"message":"Sample status message","phase":"Running"}}'
creationTimestamp: "2023-04-14T05:57:03Z"
generation: 1
name: myresource-sample-3
namespace: default
resourceVersion: "1510249"
uid: fb17b130-2f11-4388-89ee-39e1d0e23dc0
spec:
field1: value1
field2: sample field2
What environment did it happen in?
-
Crossplane version: v1.10.3
-
Kubernetes version :
Crossplane Machine: v1.18.3
Target Machine: v1.25.8 -
Kubernetes distribution:
Crossplane Machine: Minikube
Target Machine: Microk8s -
OS (e.g. from /etc/os-release)
Target Machine:
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Crossplane Machine:
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial -
Kernel (e.g.
uname -a)
Crossplane Machine: Linux sm-dev 4.4.0-124-generic #148-Ubuntu SMP Wed May 2 13:00:18 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Target Machine: Linux dev-machine 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
I believe this is the correct/expected behavior by provider-kubernetes. Typical kubernetes workflow has the user (provider-kubernetes in this case) provide the spec contents for the custom resource, which is the desired state, and the controller for the custom resource provides the status content, which is the actual state. The user does not typically provide status content, as the controller will override it with what it observes the actual state to be.
@bobh66 Thanks for the comment.
The scenario here is the custom resource(apiVersion: example.com/v1, kind: MyResource) status and spec are being updated by my custom controller (lets say controller1) and this happens in k8s cluster1 (it is not the user who is updating the spec or status). Once controller1 updates the spec and status, I am using Crossplane provider-kubernetes to push this CR from k8s cluster1 to k8s cluster2. It is expected that the provider-kubernetes creates the custom resource as in cluster2, but this is not the case here. provider-kubernetes only updates the spec section but not the status section of the custom resource in cluster2.
I assume that the MyResource controller in cluster2 (controller2) requires status information from the MyResource in cluster1, for example in a main/backup distributed application pair where the backup needs to know that the main instance is running.
You could create a MyResourceStatus resource in cluster2 with all of the information needed from MyResource cluster1 in the spec, or you could add a "remoteStatus" field in the MyResource spec and populate it with the status of the resource in cluster1 when you create it in cluster2. That would allow the controllers to maintain the status fields while still passing the status information to the remote instance.
Yes the controller2 needs the MyResource status from cluster1.
Yes I could populate it in the spec section of the CR in cluster1 and create the status in cluster2 using that info, but this would be kind of a work around/redundant field. It would be better if the provider-kubernetes could just create the CR in cluster2 with the status I pass from cluster1 in addition to spec creation. This would be a proper workflow rather than we having to do it ourselves through code.
I believe it should be feasible as it is just another call to k8s API server to update the status(in addition to the API call to update spec section).
If status.phase represents the status of the resource in cluster1 (it's an output), why doesn't it represent the status of the resource instance that is deployed in cluster2? It seems like status.phase is being treated as both an output in cluster1 and an input in cluster2? How does the controller in cluster2 know to use status.phase as an input? What happens when the controller in cluster2 overwrites status.phase with the actual status of the MyResource instance in cluster2? If this scenario was supported provider-kubernetes would be fighting with the MyResource controller in cluster2 for control of the status.phase field.
I think it makes more sense to treat status.phase consistently as an output all of the time, and define a new input to transfer the remote status as an input to the second instance.
MyResource is registered both with controller1 and controller2, so they are aware of the MyResource object and know how to process/manage it. Controller1 (in cluster1) is responsible for processing status.Phase=PHASE1 and status.Phase=PHASE2 of the MyResource CR and controllers is responsible for processing the status.Phase=PHASE3 and status.Phase=PHASE4 statuses. So, yes the output of cluster1 (status.Phase=PHASE2) acts as an input to cluster2 and controller2 is responsible for taking the MyResource further from that state (status.Phase=PHASE2).
What happens when the controller in cluster2 overwrites status.phase with the actual status of the MyResource instance in cluster2?
It does not matter. That is the expected behavior we are looking for. As mentioned earlier controller1 is responsible for PHASE1 and PHASE2 and controller2 is responsible for PHASE3 and PHASE4 of MyResource, so there is no overlap or race condition possibility as such.
If this scenario was supported provider-kubernetes would be fighting with the MyResource controller in cluster2 for control of the status.phase field.
Not really. What I am looking for is not provider-kubernetes to monitor/manage/observe the status field of MyResource, I agree that it is not the responsibility of provider-kubernetes to do that. What I want is provider-kubernetes along with creating the CR(with the spec field), it must also create the status field of the object(same status that is mentioned in composition, it does not have to create any new status, just apply the spec and status fields mentioned in the managed resource in compsition)
Any update on this?
I actually see other instances where this would be useful.
Consider examples whereby crossplane is used to reflect external resources back into the cluster for consumption by others but not as a resource to be consumed directly by the origin controller.
One example of this would be ClusterAPI and the cluster.x-k8s.io/managed-by annotation which states:
It can be applied to InfraCluster resources to signify that some external system is managing the cluster infrastructure. Provider InfraCluster controllers will ignore resources with this annotation. An external controller must fulfill the contract of the InfraCluster resource. External infrastructure providers should ensure that the annotation, once set, cannot be removed.
Furthermore, the proposed contract of the cluster.x-k8s.io/managed-by annotation goes on to clarify:
The external management system must:
- Populate all required fields within the InfraCluster spec to allow other CAPI components to continue as normal.
- Adhere to all Cluster API contracts for infrastructure providers.
- When the infrastructure is ready, set the appropriate status as is done by the provider controller today.
Within this, the last point is pertinent.
- When the infrastructure is ready, set the appropriate status as is done by the provider controller today.
In the use-case I am applying, crossplane becomes that external management system, but it in itself does not create CAPI resources so to achieve this, I'm creating the resources with this provider but in difference to (how I understand) the OPs issue, I explicitly require crossplane to manage the status; no other controller should be able to operate upon it unless I first remove control from crossplane.