Reconciliation stuck because of Sealed Secrets
jmsariron opened this issue · 3 comments
Describe the bug
We are having this issue, where, not sure how exactly, but sometimes the Flux reconciliation is failing because of Sealed Secrets, which are (As far as I know) correct.
Kustomization gets stuck in Reconciliation in Progress, getting the details I get a:
Warning HealthCheckFailed 19m kustomize-controller health check failed after 59m30.097886897s: timeout waiting for: [SealedSecret/myapp/myapp-secrets status: 'InProgress']
Checking the kustomize-controllerpod logs, it shows a server-side apply completed with all elements being unchanged and then the Reconciation Failed log message, I'm kinda confused.
Going to the SealedSecret on the Namespace it shos as correctly applied and Synced, with the corresponding regular Secret generated.
This is happening only sometimes and we can't figure out why. Since I can't reproduce it 100% of time I'm looking for some help to debug this behaviour.
Steps to reproduce
We are using SealedSecrets on other projects and AFAIK the same way, sometimes it fails sometimes not, so I don't know how to reproduce it exactly
Expected behavior
I should reconcile just fine
Screenshots and recordings
No response
OS / Distro
N/A
Flux version
N/A
Flux check
N/A
Git provider
No response
Container Registry provider
No response
Additional context
No response
Code of Conduct
- I agree to follow this project's Code of Conduct
Adding a bit more info. for this case particular I:
- Deleted the Sealed Secret inside Kubernetes
- Deleted the kustomize-controller pod
When the new kustomize-controller started syncing everything, it fixed the kustomization and now appears as Synced with the Sealed Secret recreated from the git repo, so it seems to be something related to he controller?
Hi, same issue here, restarting sealed secret pod also solve this.
Any idea ?
SealedSecret does now comply with the kstatus standard condition (Ready=true), to solve this we'll need to implement #4528 and make use of CEL to define a custom health check that looks at the Synced status.