unable to update route error
tchellomello opened this issue · 3 comments
tchellomello commented
I'm testing the cert-utils-operator on Openshift 4.9.21 and I'm hitting the error below:
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-28T18:43:58.465Z ERROR controllers.route_certificate_controller unable to update route {"route-certificate": "default/wildcard-https", "route": {"apiVersion": "route.openshift.io/v1", "kind": "Route", "namespace": "default", "name": "wildcard-https"}, "error": "Operation cannot be fulfilled on routes.route.openshift.io \"wildcard-https\": the object has been modified; please apply your changes to the latest version and try again"}
[cert-utils-operator-controller-manager-644575f449-ff67v manager] github.com/go-logr/zapr.(*zapLogger).Error
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
[cert-utils-operator-controller-manager-644575f449-ff67v manager] github.com/redhat-cop/cert-utils-operator/controllers/route.(*RouteCertificateReconciler).Reconcile
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/work/cert-utils-operator/cert-utils-operator/controllers/route/route_controller.go:213
[cert-utils-operator-controller-manager-644575f449-ff67v manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
[cert-utils-operator-controller-manager-644575f449-ff67v manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
[cert-utils-operator-controller-manager-644575f449-ff67v manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.BackoffUntil
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.JitterUntil
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.UntilWithContext
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-28T18:43:58.465Z DEBUG util.api object is not ConditionsAware, not setting status
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-28T18:43:58.465Z ERROR controller-runtime.manager.controller.route Reconciler error {"reconciler group": "route.openshift.io", "reconciler kind": "Route", "name": "wildcard-https", "namespace": "default", "error": "Operation cannot be fulfilled on routes.route.openshift.io \"wildcard-https\": the object has been modified; please apply your changes to the latest version and try again"}
[cert-utils-operator-controller-manager-644575f449-ff67v manager] github.com/go-logr/zapr.(*zapLogger).Error
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
[cert-utils-operator-controller-manager-644575f449-ff67v manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:302
[cert-utils-operator-controller-manager-644575f449-ff67v manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
[cert-utils-operator-controller-manager-644575f449-ff67v manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.BackoffUntil
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.JitterUntil
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.UntilWithContext
[cert-utils-operator-controller-manager-644575f449-ff67v manager] /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-28T18:43:58.465Z DEBUG controller-runtime.manager.events Warning {"object": {"kind":"Route","namespace":"default","name":"wildcard-https","uid":"8b51b5c7-7a20-4585-bac0-8bef898de398","apiVersion":"route.openshift.io/v1","resourceVersion":"1174098"}, "reason": "ProcessingError", "message": "Operation cannot be fulfilled on routes.route.openshift.io \"wildcard-https\": the object has been modified; please apply your changes to the latest version and try again"}
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-28T18:43:58.519Z DEBUG util.api object is not ConditionsAware, not setting status
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-28T18:43:58.519Z DEBUG util.api object is not ConditionsAware, not setting status Some of the artifacts that I'm using:
custom domain
---
apiVersion: managed.openshift.io/v1alpha1
kind: CustomDomain
metadata:
name: testing-local
spec:
domain: testing.local
certificate:
name: wildcard-testing-local-tls-cert
namespace: default
## patching
kubectl -n openshift-ingress-operator patch ingresscontroller testing-local -p '{"spec":{"routeAdmission":{"wildcardPolicy":"WildcardsAllowed"}}}' --type=merge
kubectl -n openshift-ingress-operator patch ingresscontroller default -p '{"spec":{"routeAdmission":{"wildcardPolicy":"WildcardsAllowed"}}}' --type=merge
kubectl -n openshift-ingress-operator patch ingresscontroller testing-local -p '{"spec":{"domain":"testing.local"}}' --type=merge
kubectl -n openshift-ingress-operator patch ingresscontroller testing-local -p '{"spec":{"routeSelector":{"matchLabels":{"type":"testing-local"}}}}' --type=mergecertificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: wildcard
namespace: default
spec:
secretName: wildcard-testing-local-tls-cert
duration: 2160h # 90d
renewBefore: 360h # 15d
subject:
organizations:
- Celonis
commonName: wildcard.testing.local
isCA: false
privateKey:
algorithm: RSA
encoding: PKCS1
size: 2048
usages:
- server auth
- client auth
dnsNames:
- wildcard.testing.local
- "*.testing.local"
issuerRef:
name: testing-local-ca
kind: ClusterIssuer
group: cert-manager.io
secretTemplate:
annotations:
# https://github.com/redhat-cop/cert-utils-operator#generating-kubernetes-events
cert-utils-operator.redhat-cop.io/generate-cert-expiry-alert: "true"
cert-utils-operator.redhat-cop.io/cert-expiry-check-frequency: "7d" # (days) with which frequency should the system check is a certificate is expiring
cert-utils-operator.redhat-cop.io/cert-soon-to-expire-check-frequency: "1h" # (hours) with which frequency should the system check is a certificate is expired, once it's close to expiring
cert-utils-operator.redhat-cop.io/cert-soon-to-expire-threshold: "2160h" # (90 days) what is the interval of time below which we consider the certificate close to expiry
route
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
labels:
app: hello-world
type: testing-local
name: wildcard-https
annotations:
cert-utils-operator.redhat-cop.io/inject-CA: "false"
cert-utils-operator.redhat-cop.io/certs-from-secret: "wildcard-testing-local-tls-cert"
route.openshift.io/termination: edge
spec:
host: wildcard.testing.local
port:
targetPort: 8080-tcp
tls:
termination: edge
insecureEdgeTerminationPolicy: Redirect
to:
kind: Service
name: hello-world
weight: 100
wildcardPolicy: SubdomainRunning a curl command I can see the certificate without problems, but why is the operator throwing this error?
Thanks
mmello
raffaelespazzoli commented
usually that error is transient, and it happens when two operators update an object at the same time. It's the optimistic locking mechanism inside the go-client. Retrying usually fixes the error, so I was going to ask if eventually the operator was able to put the certs in the secret.
It looks like now from the manifest you shared, but can you confirm? anything else suspect in the log.
The reason why you see the secret is that you are setting it as the ingress default (so that every route will get it unless a route specific in defined) and you are setting it as a route specific secret.
You only need it once, being it a wildcard secret, it should probably be set as the default cert in the ingress.
tchellomello commented
thanks for your quick response @raffaelespazzoli
yes, the cert-manager operator created and populate the wildcard-testing-local-tls-cert secret and then which was later consumed by the route. If I inspect the route object, I can see the cert-utils-operator
kubectl get routes wildcard-https -o yaml | kubectl neat
apiVersion: route.openshift.io/v1
kind: Route
metadata:
annotations:
cert-utils-operator.redhat-cop.io/certs-from-secret: wildcard-testing-local-tls-cert
cert-utils-operator.redhat-cop.io/inject-CA: "false"
route.openshift.io/termination: edge
labels:
app: hello-world
type: testing-local
name: wildcard-https
namespace: default
spec:
host: wildcard.testing.local
port:
targetPort: 8080-tcp
tls:
certificate: |
-----BEGIN CERTIFICATE-----
XXXX
-----END CERTIFICATE----
insecureEdgeTerminationPolicy: Redirect
key: |
-----BEGIN RSA PRIVATE KEY-----
xxxxx
-----END RSA PRIVATE KEY-----
termination: edge
to:
kind: Service
name: hello-world
weight: 100
wildcardPolicy: SubdomainI also can confirm that whenever I renew the certificate via cert-manager, the operator does it job updating the secret.
For example:
before
$ curl -k -L -v https://b.testing.local 2>&1 >/dev/null | grep 'date:' | head -n2
* start date: Mar 28 18:46:39 2022 GMT
* expire date: Jun 26 18:46:39 2022 GMTrefresh cert
$ kubectl cert-manager renew -n default wildcard
Manually triggered issuance of Certificate default/wildcardafter
$ curl -k -L -v https://b.testing.local 2>&1 >/dev/null | grep 'date:' | head -n2
* start date: Mar 29 14:36:38 2022 GMT
* expire date: Jun 27 14:36:38 2022 GMTUpon refresh, the cert-utils-operator logs seem to be fine
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-29T14:44:52.272Z DEBUG util.api object is not ConditionsAware, not setting status
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-29T14:44:52.272Z DEBUG util.api object is not ConditionsAware, not setting status
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-29T14:45:10.923Z DEBUG controller-runtime.manager.events Warning {"object": {"kind":"Secret","namespace":"default","name":"wildcard-testing-local-tls-cert","uid":"2c0cc2b1-f326-48e6-b0f2-9422857cd3cc","apiVersion":"v1","resourceVersion":"1553226"}, "reason": "Certs Soon to Expire", "message": "Certificate expiring in 89 days"}
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-29T14:45:10.989Z DEBUG util.api object is not ConditionsAware, not setting status
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-29T14:45:10.990Z DEBUG util.api object is not ConditionsAware, not setting status
So @raffaelespazzoli is it safe and expected then to assume the race condition error?
Thanks
tchellomello commented
Thanks, @raffaelespazzoli for your assistance.
I'm closing this issue.