redhat-cop/namespace-configuration-operator

can't delete resource - CR stuck

davidkarlsen opened this issue · 9 comments

davidkarlsen commented

I tried deleting a NamespaceConfiguration resource, but it won't be deleted due to a stuck finalizer, and the operator tries updating the object with a finalizer, which is not allowed because it is being deleted

2021-09-09T14:53:26.583Z        ERROR   controllers.NamespaceConfig     unable to update instance       {"namespaceconfig": "/fss-apps", "instance": {"apiVersion": "redhatcop.redhat.io/v1alpha1", "kind": "NamespaceConfig", "name": "fss-apps"}, "error": "NamespaceConfig.redhatcop.redhat.io \"fss-apps\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"namespaceconfig-controller\"}"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
github.com/redhat-cop/namespace-configuration-operator/controllers.(*NamespaceConfigReconciler).Reconcile
        /workspace/controllers/namespaceconfig_controller.go:85
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
raffaelespazzoli commented

can you paste the instance(yaml)?
There should always ever be one finalizer. And the logic should be: if the instance is new, the finalizer is added, if the instance is being delete and the clean up has been completed the finalizer is removed...

davidkarlsen commented 
--- a/manifests/governance/namespace-configurator/rolebindings.yaml
+++ /dev/null
@@ -1,44 +0,0 @@
-apiVersion: redhatcop.redhat.io/v1alpha1
-kind: NamespaceConfig
-metadata:
-  name: fss-apps
-spec:
-  labelSelector:
-    matchLabels:
-      app.kubernetes.io/managed-by: clout
-      fss.tietoevry.com/legacy-logging: "true"
-  templates:
-  - objectTemplate: |
-      apiVersion: rbac.authorization.k8s.io/v1
-      kind: RoleBinding
-      metadata:
-        name: fss-apps
-        namespace: {{ .Name }}
-        labels:
-          app.kubernetes.io/managed-by: namespace-configuration-operator
-      roleRef:
-        apiGroup: rbac.authorization.k8s.io
-        kind: ClusterRole
-        name: fss-apps
-      subjects:
-      - apiGroup: rbac.authorization.k8s.io
-        kind: Group
-        name: system:serviceaccounts:{{ .Name }}
raffaelespazzoli commented
davidkarlsen commented

sorry, that's gone by now.
I'll reopen a case if I can reproduce.

davidkarlsen commented

@raffaelespazzoli found the cause, the finalizers seems to have changed name at some point, leading to having two:

finalizers:
  - namespace-config-operator
  - namespaceconfig-controller
raffaelespazzoli commented
davidkarlsen commented

this should be the right one: "namespaceconfig-controller". Also can you share between which version you see the change, for the benefit of others. And sorry for the inconvenience.
…
On Tue, Oct 19, 2021 at 2:54 PM David J. M. Karlsen < @.***> wrote: @raffaelespazzoli <@raffaelespazzoli> found the cause, the finalizers seems to have changed name at some point, leading to having two: finalizers: - namespace-config-operator - namespaceconfig-controller — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#119 (comment)>, or unsubscribe <github.com/notifications/unsubscribe-auth/ABPERXADNOBC5DJICEHW2LTUHW5EVANCNFSM5DXLARPQ> . Triage notifications on the go with GitHub Mobile for iOS <apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
-- ciao/bye Raffaele

I don't know in which version, I've upgraded several times.

davidkarlsen commented

Maybe a fix could be made in the controller to look for the invalid one and simply remove it?

raffaelespazzoli commented