Issue on authenticate ingress endpoint

[jluque0101]

What happened?

pomerium-authenticate ingress has UpdateError, it can't get IP therefore authenticate.mydomain.com endpoint is not available.

 Events:
  Type     Reason       Age                   From              Message
  ----     ------       ----                  ----              -------
  Warning  UpdateError  7m38s (x18 over 13m)  pomerium-ingress  get config: get pomerium config: rpc error: code = Internal desc = cryptutil: decryption failed (mismatched keys?): chacha20poly1305: message authentication failed
  Warning  UpdateError  12s (x18 over 5m39s)  pomerium-ingress  get config: get pomerium config: rpc error: code = Internal desc = cryptutil: decryption failed (mismatched keys?): chacha20poly1305: message authentication failed

What did you expect to happen?

Steps to reproduce

1. Deleted previous *-pomerium secrets

• pomerium
• pomerium-authenticate-tls
• pomerium-authenticate-token-tgsw8
• pomerium-authorize-tls
• pomerium-authorize-token-xxmkp
• pomerium-ca-tls
• pomerium-config
• pomerium-databroker-tls
• pomerium-databroker-token-8h6wb
• pomerium-ingress-controller-token-wcswp
• pomerium-proxy-token-nzgdm
• pomerium-redis-password
• pomerium-redis-tls
• pomerium-redis-token-4t8vz
• pomerium-shared
• pomerium-signing-key

3. Installed chart with values from below
4. Run kubectl describe ingress pomerium-authenticate

What's your environment like?

• Chart version: 31.0.2
• Container image:
• Kubernetes version: 1.21
• Cloud provider:
• Other details:

What are your chart values?

    config:
      rootDomain: mydomain.com
    authenticate:
      idp:
        provider: okta
        url: https://myokta.okta.com
    forwardAuth:
      enabled: true
    redis:
      enabled: true
    ingressController:
      enabled: true
      ingressClassResource:
        defaultCertSecret: ns/default-tls
    ingress:
      enabled: false
    extraEnv:
      LOG_LEVEL: info
    extraEnvFrom:
      - secretRef:
          name: pomerium-config
    resources:
      limits:
        cpu: 200m
        memory: 500Mi
      requests:
        cpu: 100m
        memory: 300Mi
    priorityClassName: ""
    metrics:
      enabled: true
    serviceMonitor:
      enabled: true

What are the contents of your config secret?

kubectl get secret pomerium -o=jsonpath="{.data['config\.yaml']}" | base64 -D

autocert: false
dns_lookup_family: V4_ONLY
address: :443
grpc_address: :443
certificate_authority_file: "/pomerium/ca/ca.crt"
certificates:
authenticate_service_url: https://authenticate.mydomain.com
authorize_service_url: https://pomerium-authorize.ingress.svc.cluster.local
databroker_service_url: https://pomerium-databroker.ingress.svc.cluster.local
idp_provider: okta
idp_scopes: 
idp_provider_url: https://myokta.okta.com
forward_auth_url: https://forwardauth.mydomain.com
idp_client_id: REPLACE_ME
idp_client_secret: REPLACE_ME
databroker_storage_tls_skip_verify: false    
routes:

What did you see in the logs?

# Paste your logs here.
# Be sure to scrub any sensitive values
pomerium-authenticate-684f948648-mvs5s pomerium {"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","time":"2022-03-30T10:45:07Z","message":"initial sync"}
pomerium-authenticate-684f948648-mvs5s pomerium {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"rpc error: code = Unknown desc = cryptutil: decryption failed (mismatched keys?): chacha20poly1305: message authentication failed","time":"2022-03-30T10:45:07Z","message":"error during initial sync"}
pomerium-authenticate-684f948648-mvs5s pomerium {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"rpc error: code = Unknown desc = cryptutil: decryption failed (mismatched keys?): chacha20poly1305: message authentication failed","time":"2022-03-30T10:45:07Z","message":"sync"}
pomerium-authenticate-684f948648-mvs5s pomerium {"level":"info","service":"envoy","name":"main","time":"2022-03-30T10:46:02Z","message":"shutting down parent after drain"}

--------

pomerium-proxy-5cc54748f4-6m2gq pomerium {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"rpc error: code = Unknown desc = cryptutil: decryption failed (mismatched keys?): chacha20poly1305: message authentication failed","time":"2022-03-30T10:46:22Z","message":"sync"}

Additional context

I am wondering if I might have leftovers from previous pomerium installs other than secrets mentioned.

[jluque0101]

My problem was related to Redis PVCs, after removing those everything worked on for a fresh install.
Maybe is worth to mention at uninstalling the chart section.

Thanks!