Ingress missing necessary Nginx annotation

[fightingsleep]

Following the latest commit to ingress.yaml, this chart no longer plays nicely with the Kubernetes flavor of the Nginx ingress controller.

The ingress controller will produce logs like:

"Ignoring ingress because of error while validating ingress class" ingress="default/docker-registry" error="no object matching key "nginx"

This is caused by the missing kubernetes.io/ingress.class: "nginx" annotation on the ingress created from the ingress.yaml template.

We can no longer add that annotation due to the fact that you'll get the following error during helm install docker-registry twuni/docker-registry -f docker-registry-config.yaml (Note: see docker-registry-config.yaml down below):

Error: INSTALLATION FAILED: Ingress.extensions "docker-registry" is invalid: annotations.kubernetes.io/ingress.class: Invalid value: "nginx": can not be set when the class field is also set

This is due to the recent addition of ingressClassName: {{ .Values.ingress.className }} in ingress.yaml.

Steps to reproduce:

Setup the Nginx controller

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace

Setting up cert manager

helm repo add jetstack https://charts.jetstack.io
helm repo update
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.crds.yaml
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.9.1
kubectl apply -f cert-issuer.yaml
kubectl apply -f certificate.yaml

where cert-issuer.yaml is

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod-clusterissuer
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: some@email.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-secret-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx

and certificate.yaml is

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: docker-registry-certificate
spec:
  secretName: letsencrypt-secret-prod
  duration: 2160h
  renewBefore: 360h
  issuerRef:
    name: letsencrypt-prod-clusterissuer
    kind: ClusterIssuer
  dnsNames:
  - some.hostname.com

Set up the docker registry

helm repo add twuni https://helm.twun.io
helm repo update
helm install docker-registry twuni/docker-registry -f docker-registry-config.yaml

where docker-registry-config.yaml is

ingress:
  enabled: true
  hosts:
    - some.hostname.com
  annotations:
    # THIS ANNOTATION IS NECESSARY, BUT NOT ALLOWED: kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: letsencrypt-prod-clusterissuer
  tls:
    - secretName: letsencrypt-secret-prod
      hosts:
      - some.hostname.com
storage: s3
secrets:
  htpasswd: |-
    <USERNAME>:<ENCRYPTED PASSWORD>
  s3:
    accessKey: "<ACCESS KEY>"
    secretKey: "<SECRET>"
s3:
  region: eu-central-1
  regionEndpoint: eu-central-1.<SOME OBJECT STORAGE HOST>.com
  secure: true
  bucket: <SOME BUCKET>

This is easy enough to work-around (see below). However, this used to work fine out of the box. Here is an example video of what the setup used to be like using older versions of everything.

Click drop-down for workaround

You can workaround this by setting enabled: false in docker-registry-config.yaml, then creating a separate ingress yourself like so:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: docker-registry
  namespace: default
  labels:
    app: docker-registry
    chart: docker-registry-2.2.2
    release: docker-registry
    heritage: Helm
  annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod-clusterissuer"
      kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - host: host.name.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: docker-registry
                port:
                  number: 5000
  tls:
    - hosts:
      - host.name.com
      secretName: letsencrypt-secret-prod

[canterberry]

Thank you for the detailed report!

It sounds like there are conflicting implementations of an annotation-driven ingress controller, with different and mutually exclusive requirements. This chart currently works with one, but not the other.

If that interpretation is correct, then the simplest option seems to be to add a configuration value for which kind of ingress controller (vendor/version/whatever the distinguishing factor is between the two) is in use -- that would drive which of the two mechanisms is present (ingressClassName field vs annotation).

If that interpretation is not correct, then could you distill the issue down to the (metaphorical) line where it works on one side and doesn't on the other? I don't use ingresses, myself, so it's nice to learn these details from someone who does.

[canterberry]

Follow-up: In the repo you linked for the Kubernetes ingress controller, their own example for deploying a docker registry shows use of the ingressClassName field: https://github.com/kubernetes/ingress-nginx/blob/main/docs/examples/docker-registry/ingress-without-tls.yaml . I'm wondering if maybe there is potentially something custom about your ingress controller installation that would cause the first error you shared (i.e: it can't find an nginx implementation for the ingressClassName).

[fightingsleep]

Your interpretation is correct

🤔 There shouldn't be anything special with the ingress controller install. It's running on a fresh cluster, and I'm using the helm commands from the Setup the Nginx controller section above.

Here is the result:

The chart version and app version are quite recent.