CRDs are not installed on Operator installation

[ydixken]

Describe the bug

Since a couple of weeks the operator does not install its respective CRDs (except for nginxingresses.charts.nginx.org) on our OpenShift clusters anymore, I am able to reproduce this with our internal tooling (that is setting up the CatalogSource, Subscription and so on..) and also with the manual installation described here.

For the manual installation that's the log:

[ydixken@silence.local:redacted/redacted/nginx-ingress-helm-operator] » make deploy IMG=nginx/nginx-ingress-operator:1.5.0

cd config/manager && /Users/ydixken/development/redacted/redacted/nginx-ingress-helm-operator/bin/kustomize edit set image controller=nginx/nginx-ingress-operator:1.5.0
/Users/ydixken/development/redacted/redacted/nginx-ingress-helm-operator/bin/kustomize build config/default | kubectl apply -f -
namespace/nginx-ingress-operator-system created
Warning: resource customresourcedefinitions/nginxingresses.charts.nginx.org is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/nginxingresses.charts.nginx.org configured
serviceaccount/nginx-ingress-operator-controller-manager created
role.rbac.authorization.k8s.io/nginx-ingress-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-operator-manager-role created
Warning: resource clusterroles/nginx-ingress-operator-metrics-reader is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
clusterrole.rbac.authorization.k8s.io/nginx-ingress-operator-metrics-reader configured
Warning: resource clusterroles/nginx-ingress-operator-nginx-ingress-admin is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
clusterrole.rbac.authorization.k8s.io/nginx-ingress-operator-nginx-ingress-admin configured
clusterrole.rbac.authorization.k8s.io/nginx-ingress-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-operator-proxy-rolebinding created
service/nginx-ingress-operator-controller-manager-metrics-service created
deployment.apps/nginx-ingress-operator-controller-manager created

Actual Behaviour

11:03 $ oc get crds | grep -e 'nginx'

nginxingresses.charts.nginx.org                                        2023-07-11T08:55:37Z

Expected after the installation:

11:17 $ oc get crds | grep -e 'f5\|nginx'

apdoslogconfs.appprotectdos.f5.com                                     2023-07-11T09:17:38Z
apdospolicies.appprotectdos.f5.com                                     2023-07-11T09:17:39Z
aplogconfs.appprotect.f5.com                                           2023-07-11T09:17:38Z
appolicies.appprotect.f5.com                                           2023-07-11T09:17:38Z
apusersigs.appprotect.f5.com                                           2023-07-11T09:17:38Z
dnsendpoints.externaldns.nginx.org                                     2023-07-11T09:17:39Z
dosprotectedresources.appprotectdos.f5.com                             2023-07-11T09:17:39Z
globalconfigurations.k8s.nginx.org                                     2023-07-11T09:17:39Z
nginxingresses.charts.nginx.org                                        2023-07-11T08:55:37Z
policies.k8s.nginx.org                                                 2023-07-11T09:17:39Z
transportservers.k8s.nginx.org                                         2023-07-11T09:17:39Z
virtualserverroutes.k8s.nginx.org                                      2023-07-11T09:17:39Z
virtualservers.k8s.nginx.org                                           2023-07-11T09:17:39Z

This behavior might have been triggered through a failed upgrade, that I might not be aware of, I'm currently working around this issue by installing the CRDs manually. From my point of view the operator installation is succeeding correctly, with a healthy CatalogSource and a healthy OP install.

Is this an already known behaviour or changed installation workflow, that I didn't realized yet?
I can happily provide logs / conf, if needed.

Expected behavior
I expect the operator to install the CRDs, as it was doing before.

Your environment

• Version of the NGINX Ingress Operator - release version or a specific commit
  1.5.0
• Version of the Ingress Controller - release version or a specific commit
  3.2.0
• Version of Kubernetes
  OpenShift 4.12.22
• Kubernetes platform (e.g. Mini-kube or GCP)
  AWS
• Using NGINX or NGINX Plus
  We're under NGINX Plus subscriptions.

Additional context
Add any other context about the problem here. Any log files you want to share.

[ydixken]

Closing, found the issue on our end.

[brianehlert]

Would you mind sharing the resolution to help inform anyone that discovers this issue?

[ydixken]

TL;DR Please review your CRD dependencies, if you're hitting the same issue.

We have a set of self-developed tools that set up our clusters with the necessary NGINX manifests such as Ingress, ClusterRoles, APPolicies, and more. These scripts ensure that the required CRDs are present before proceeding to the next set of manifests that rely on these CRDs.

However, we encountered a potential future issue when onboarding more features than just plain NGINX IC. Therefore we've introduced a check for 'f5' and 'appprotect' CRDs (aside from previously only 'nginx\.org') as a hook, and we made the availability of these CRDs mandatory for the Ingress (!). The problem is that the Ingress itself - or its chart (which only requires nginxingresses.charts.nginx.org to be set up) is the one responsible for adding the missing CRDs.

The issue started occurring after we purged the entire operator, namespace, related resources, and removed the actual CRDs.

What we did:

1. Carefully reviewed our CRD dependencies to understand the root cause.
2. Placed the checks for 'f5' and 'appprotect' CRDs in the appropriate manifests.
3. Removed any unnecessary dependencies that were identified.