redhat-cop/namespace-configuration-operator

Namespace operator pod is crashlooping after upgrade

A749999 opened this issue · 6 comments

A749999 commented

Hi Team,

We have update the namespace configuration operator from 1.2.4 to 1.2.5 and observed the operator pod was keep getting crashlooped after upgrade and also getting the below error in manager container logs. please help us to resolve this issue.

2024-01-20T12:11:43Z INFO Observed a panic in reconciler: interface conversion: validation.Schema is *validation.schemaValidation, not *validation.NullSchema {"controller": "namespaceconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "NamespaceConfig", "NamespaceConfig": {"name":"ns-sa-ctl"}, "namespace": "", "name": "ns-sa-ctl", "reconcileID": "45df894e-e13d-4195-a0b7-f915c06bb869"}
panic: interface conversion: validation.Schema is *validation.schemaValidation, not *validation.NullSchema [recovered]
panic: interface conversion: validation.Schema is *validation.schemaValidation, not *validation.NullSchema

goroutine 186 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:115 +0x1e5
panic({0x22be620?, 0xc0072c0fc0?})
/opt/hostedtoolcache/go/1.21.4/x64/src/runtime/panic.go:914 +0x21f
github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*LockedResourceManager).validateLockedResources(0xc000724700, {0xc00072b000, 0x62, 0xc000457a60?})
/home/runner/go/pkg/mod/github.com/redhat-cop/operator-utils@v1.3.7/pkg/util/lockedresourcecontroller/locked-resource-manager.go:341 +0x6f3
github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*LockedResourceManager).SetResources(0xc000724700, {0xc00072b000?, 0x62, 0x80})
/home/runner/go/pkg/mod/github.com/redhat-cop/operator-utils@v1.3.7/pkg/util/lockedresourcecontroller/locked-resource-manager.go:82 +0x77
github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*LockedResourceManager).Restart(0xc000724700, {0x27c5e68, 0xc0007d8300}, {0xc00072b000, 0x62, 0x80}, {0x3677360, 0x0, 0x0}, 0x0, ...)
/home/runner/go/pkg/mod/github.com/redhat-cop/operator-utils@v1.3.7/pkg/util/lockedresourcecontroller/locked-resource-manager.go:222 +0x16c
github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*EnforcingReconciler).UpdateLockedResourcesWithRestConfig(0xc00016afd0, {0x27c5e68, 0xc0007d8300}, {0x27db750?, 0xc0008d4000?}, {0xc00072b000, 0x62, 0x80}, {0x3677360, 0x0, ...}, ...)
/home/runner/go/pkg/mod/github.com/redhat-cop/operator-utils@v1.3.7/pkg/util/lockedresourcecontroller/enforcing-reconciler.go:117 +0x3bc
github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*EnforcingReconciler).UpdateLockedResources(...)
/home/runner/go/pkg/mod/github.com/redhat-cop/operator-utils@v1.3.7/pkg/util/lockedresourcecontroller/enforcing-reconciler.go:91
github.com/redhat-cop/namespace-configuration-operator/controllers.(*NamespaceConfigReconciler).Reconcile(0xc00016afd0, {0x27c5e68, 0xc0007d8300}, {{{0x0, 0x0}, {0xc00057f0f0, 0x9}}})
/home/runner/work/namespace-configuration-operator/namespace-configuration-operator/controllers/namespaceconfig_controller.go:121 +0x83c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x27c5e68?, {0x27c5e68?, 0xc0007d8300?}, {{{0x0?, 0x21c61c0?}, {0xc00057f0f0?, 0x27b66d0?}}})
/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:118 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00037d220, {0x27c5ea0, 0xc0001fcd20}, {0x233bc20?, 0xc0000655c0?})
/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:314 +0x365
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00037d220, {0x27c5ea0, 0xc0001fcd20})
/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:265 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:226 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 73
/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:222 +0x565

Thanks
Siva M

raffaelespazzoli commented
A749999 commented

Thanks for your reply.. Please find the CR details which causing the issue.

apiVersion: redhatcop.redhat.io/v1alpha1
kind: NamespaceConfig
metadata:
creationTimestamp: "2021-09-28T11:02:48Z"
finalizers:

  • namespaceconfig-controller
    generation: 5
    name: ns-sa-ctl
    resourceVersion: "1593732342"
    uid: 602f466e-fe17-425d-a85e-9b09cfb5bf6e
    spec:
    annotationSelector: {}
    labelSelector:
    matchLabels:
    nsctl: "true"
    templates:
  • excludedPaths:
    • .metadata
    • .status
    • .spec.replicas
      objectTemplate: |
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
      name: {{ .Name }}-sarole
      namespace: {{ .Name }}
      roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: sa-role
      subjects:
      • kind: ServiceAccount
        name: test-sa-namespace
        namespace: itt-dev
      • kind: ServiceAccount
        name: test-sa-namespace
        namespace: ocp-dev
  • excludedPaths:
    • .metadata
    • .status
    • .spec.replicas
      objectTemplate: |
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
      name: {{ .Name }}-authn-k8s-sa-oc-3-role
      namespace: {{ .Name }}
      roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: system:public-info-viewer
      subjects:
      • kind: ServiceAccount
        name: authn-k8s-sa-oc-3
        namespace: conjur-follower-namespace
        status:
        conditions:
  • lastTransitionTime: "2024-01-20T03:36:22Z"
    message: ""
    observedGeneration: 5
    reason: LastReconcileCycleSucceded
    status: "True"
    type: ReconcileSuccess
  • lastTransitionTime: "2023-09-14T10:54:51Z"
    message: 'Get "https://172.30.0.1:443/openapi/v2?timeout=32s": net/http: request
    canceled (Client.Timeout exceeded while awaiting headers)'
    observedGeneration: 5
    reason: LastReconcileCycleFailed
    status: "True"
    type: ReconcileError
A749999 commented

Any update on this issue?

raffaelespazzoli commented

that issue should be solved in v1.2.6, can you try it?

A749999 commented

that issue should be solved in v1.2.6, can you try it?

Great.. issue was resolved after we upgrade the operator to v1.2.6

A749999 commented

issue was resolved after we upgrade the operator to v1.2.6