Validating webhook does not pick up new certificate when renewed by certificate manager
mshanmu opened this issue · 2 comments
https://github.com/flant/shell-operator/blob/main/pkg/webhook/server/server.go#L26
See above link, it looks like the webhook server loads the certificate file only once when the server starts and does not have any mechanism to pick up the new certificate and keep using the old (now invalidated) cert. Once it gets restarted the server picks up the correct certificate and things works again.
You could try using a certwatcher to fix this issue. For an example, you can check this PR kubeflow/kubeflow#6581
Expected behavior (what you expected to happen):
Works well even when certificate is changed
Actual behavior (what actually happened):
Saw the below error,
"failed to call webhook: post "":
x509: certificate has expired or not yet valid
Steps to reproduce:
- Install cert-manager in k8s
- Use self signed certificate with 5 days validity
- After 5 days, notice the " x509: certificate has expired or not yet valid" failure
Environment:
- Shell-operator version: v1.0.12
- Kubernetes version: v1.24.2
- Installation type (kubectl apply, helm chart, etc.): helm chart
Anything else we should know?:
Additional information for debugging (if necessary):
Hook script
Logs
@mshanmu I confirm that this issue is valid. It will take us some time to fix, because it is not fully aligned with our current roadmap. Yet we are always willing to accept contributions!
Thanks @nabokihms !! Will try to send in a PR for this issue.