Hot reload of mtls cert is broken. KIC can't fetch secret.

[codingwwws]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Hi
(I think this might be similar to Kong/charts#958 but not sure)

I am using a cert for mtls inside a Service resource with:

  annotations:
    konghq.com/client-cert: my-service-secret
    konghq.com/protocol: https

Im getting the error:

level=error msg="resource processing failed: failed to fetch secret 'my-service/my-service-secret': Secret my-service/my-service-secret not found" GVK="/v1, Kind=Service" name=my-service namespace=my-service

The Secret is there in the cluster and things are working fine. However once is deleted and recreated (renewed) by cert manager, the Ingress controller goes into the above error and the upstream service that is expecting the cert is receiving none.

Steps to solve currently is to add a dummy change to some annotation to the Service resource. Then something at the service level clean reloads the cert and the error goes away and the cert is reloaded correctly and sent to upstream. Or to restart the Kong and kong ingress controller pods. That also loads the new cert and the error is only in case of the hot reload scenario.

In effect the hot reload of this cert is broken.
KIC 2.12.3
Kong 3.4.2

Expected Behavior

No errors supposed to happen upon the the cert secret being renewed and recreated. This issue is not present in KIC 2.9.3 and I can consistently reproduce the same scenario with no issues.

Steps To Reproduce

1- Setup a working mtls cert 
2- Delete the secret containing the cert 
3- cert manager will renew the mtls cert, recreating the Secret (same name, just new uid and resourceVersion and the cert content is changed)
4- The above error starts and that route to that Service doesn't send the new mtls cert to the upstream

Kong Ingress Controller version

2.12.3

Kubernetes version

1.28.6

Anything else?

No response