Field is immutable

[jicowan]

When trying to create a space as a user (not cluster-admin) I get the following error:

$ cat << 'EOF' | kubectl apply --kubeconfig joe_bloggs_config -f -
apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Space
metadata:
  name: joes-space
EOF

The Space "joes-space" is invalid: metadata.annotations: Invalid value: map[string]string{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"tenancy.kiosk.sh/v1alpha1\",\"kind\":\"Space\",\"metadata\":{\"annotations\":{},\"name\":\"joes-space\"}}\n"}: field is immutable, try updating the namespace

kubectl auth can-i --kubeconfig joe_bloggs_config create Space
Warning: resource 'spaces' is not namespace scoped in group 'tenancy.kiosk.sh'
yes

[LukasGentele]

@jicowan Thanks for opening this issue. Seems like a bug. To narrow it down: Can you try kubectl create instead of kubectl apply? Does that work?

[jicowan]

create works.

[FabianKramm]

@jicowan thanks for reporting, yes this was initially implemented to prevent users from overwriting annotations and labels defined in the space template in their account. kubectl create works because it is not setting any annotations, while kubectl apply does (for the three way merge later on). However, I changed this behavior so now that when annotations and labels are defined in the space template they will overwrite the ones provided by the user otherwise it is allowed to set annotations and labels and they will just go through.

[jicowan]

@FabianKramm there is a similar effort underway in sig-multitenant with the tenant CRD and hierarchical namespace controller. Are you working with the community on this effort?

[FabianKramm]

@jicowan yes we are very aware of the tenant CRD and hierarchical namespace controller. While I think parts of the scope and goals of these projects are aligned with what kiosk is trying to accomplish, there are also quite some differences.

The basic primitive in the approach of the multi tenancy group is a tenant crd object and the corresponding tenant admin namespace that comprises all related resources to that tenant, such as allowed namespaces (tenant namespace crd), roles etc. The hierarchical namespace controller allows to define sync hierarchies between namespaces and basically allows some sort of templating for new and existing namespaces. When we first thought about kiosk and what it should accomplish we initially wanted to build upon these efforts from the multi-tenancy group, however we had some concerns with the approach the multi tenancy took:

• There were several problems from an users endpoint we wanted to improve on: for example, retrieving allowed namespaces an user has access to is not trivial with the approach the multi-tenancy group took, since the user at least needs to know its tenant admin namespace. In kiosk, you can easily run kubectl get spaces or kubectl get accounts as any user and see only the resources you have access to. This approach is more like how openshift is doing things with projects, which we believe is the superior approach instead of creating access resources (tenant namespaces) in each tenant admin namespace which takes additional space in the etcd.
• There is an overhead in unused resources that are just there for templating and defining hierarchies but are actually functional in kubernetes and controllers will act on them: each tenant has one admin namespace and potentially additional resources (like resource quotas, network policies etc.) that will be synced to tenant namespaces. These resources are all functional itself and could slow down kubernetes controllers since caches get bigger and those resources also need to be reconciled. We feel this overhead is not needed and template resources should be separated from "real" resources.
• Since the kubernetes sigs are a place to develop future standards for certain areas, we felt the pace these projects were moving for now was not fast enough for us. There are several efforts in the multi tenancy sig on the way, however we still feel that they are not mature enough to be built on. We are very aware that might change in future and we are very open to replace parts of kiosk with efforts from the multi tenancy group if they reach a more mature level and are widely accepted. We are also very open to help them in certain areas.

These concerns lead to the decision to build kiosk not on top of the efforts of the multi tenancy group for now. Nonetheless, these initial problems we had with their approach does not mean their efforts and our efforts are mutually exclusive and we might reconsider in the future. Besides the technical foundation, I think kiosk has a wider scope than what the multi tenancy group currently wants to build: e.g. multi cluster management. So I think kiosk is definitely not designed as a competitor to what the multi tenancy group is doing, it is rather designed as a project that provides a single coherent (though opinionated) solution to the topic that might be build on some of the building blocks the multi tenancy group provides in the future.

[jicowan]

@FabianKramm Thanks for the very thorough explanation. I like the CRD approach because they can be scoped to a namespace which I assume allows Kiosk to give tenants visibility to the resources, e.g. accounts and tenants, they "own". It really wasn't clear from reading the KEP for the hierarchical namespace controller which resources would be synchronized with child namespaces. I think it mentioned RBAC and network policies but then the tenant KEP seemed to imply that you could define other objects. We should probably continue this discussion in another issue/thread since it no longer pertains to "field is immutable" issue I was having. Thanks again for the response!