nabsul/kcert

Certificate renewal failed

koehoorn opened this issue · 3 comments

Hi,
i dont know how to debug this further.
my setup is arm64 ( pine64 boards) running ubuntu and microk8s with the metallb software load balancer leading into ingress-nginx controller. then i created an nginx 'hello world' pod with persistent storage and a service and an ingress, so far works well on http.

Now i want to add tls termination on the ingress controller and found your project. this looks much cleaner then cert-manager.
so i defined the label and the tls section in the ingress definition and i see kcert responding. it tries to create secrets and get it signed, but that last part seems to fail. ''renewal failed", as you can see from the error logs below.

  • tried production url too.
  • tried longer ACME__VALIDATIONWAITTIMESECONDS 60

as soon as i define a tls section in the ingress, i get a certificate from ca 'Kubernetes Ingress Controller Fake Certificate' and it redirects http into https. i cant see my hello world message on http anymore. (perhaps thats why the challenge is unaccessible?)
i am guessing it is unable to host the challenge token on http. How does this part work and how to trace and debug this?

kubectl logs -n kcert kcert-56c78dc7cc-949hq
{"EventId":0,"LogLevel":"Information","Category":"KCert.Services.RenewalHandler","Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","State":{"Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","authUri":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283","status":"invalid","{OriginalFormat}":"Get Auth {authUri}: {status}"}} {"EventId":0,"LogLevel":"Information","Category":"KCert.Services.RenewalHandler","Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","State":{"Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","authUri":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283","status":"invalid","{OriginalFormat}":"Get Auth {authUri}: {status}"}} {"EventId":0,"LogLevel":"Information","Category":"KCert.Services.RenewalHandler","Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","State":{"Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","authUri":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283","status":"invalid","{OriginalFormat}":"Get Auth {authUri}: {status}"}} {"EventId":0,"LogLevel":"Information","Category":"KCert.Services.RenewalHandler","Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","State":{"Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","authUri":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283","status":"invalid","{OriginalFormat}":"Get Auth {authUri}: {status}"}} {"EventId":0,"LogLevel":"Information","Category":"KCert.Services.RenewalHandler","Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","State":{"Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","authUri":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283","status":"invalid","{OriginalFormat}":"Get Auth {authUri}: {status}"}} {"EventId":0,"LogLevel":"Error","Category":"KCert.Services.RenewalHandler","Message":"Certificate renewal failed.","Exception":"System.Exception: Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283 did not complete in time. Last Response: invalid at KCert.Services.RenewalHandler.ValidateAuthorizationAsync(String key, String kid, String nonce, Uri authUri) in /build/Services/RenewalHandler.cs:line 111 at KCert.Services.RenewalHandler.RenewCertAsync(String ns, String secretName, String[] hosts) in /build/Services/RenewalHandler.cs:line 45","State":{"Message":"Certificate renewal failed.","{OriginalFormat}":"Certificate renewal failed."}} {"EventId":0,"LogLevel":"Error","Category":"KCert.Services.KCertClient","Message":"Renewal failed","Exception":"KCert.Models.RenewalException: Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283 did not complete in time. Last Response: invalid ---\u003E System.Exception: Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283 did not complete in time. Last Response: invalid at KCert.Services.RenewalHandler.ValidateAuthorizationAsync(String key, String kid, String nonce, Uri authUri) in /build/Services/RenewalHandler.cs:line 111 at KCert.Services.RenewalHandler.RenewCertAsync(String ns, String secretName, String[] hosts) in /build/Services/RenewalHandler.cs:line 45 --- End of inner exception stack trace --- at KCert.Services.RenewalHandler.RenewCertAsync(String ns, String secretName, String[] hosts) in /build/Services/RenewalHandler.cs:line 57 at KCert.Services.KCertClient.RenewCertAsync(Task prev, String ns, String secretName, String[] hosts, CancellationToken tok) in /build/Services/KCertClient.cs:line 63","State":{"Message":"Renewal failed","{OriginalFormat}":"Renewal failed"}} {"EventId":0,"LogLevel":"Information","Category":"KCert.Services.EmailClient","Message":"Cannot send email email because it\u0027s not configured correctly","State":{"Message":"Cannot send email email because it\u0027s not configured correctly","{OriginalFormat}":"Cannot send email email because it\u0027s not configured correctly"}}

microk8s uses 'ingress.class public', found via
https://stackoverflow.com/questions/54506269/simple-ingress-from-host-with-microk8s
so i tried to set kubernetes.io/ingress.class: public in the ingress definition, but same effect.

okay i rubberducked this aperantly. ;)

i assumed you used/assumed the ingress.class annotation to be nginx in your code. so i changed the ingress controller daemonset to define nginx.

kubectl edit ds nginx-ingress-microk8s-controller -n ingress

`

  • args:
    - /nginx-ingress-controller
    - --configmap=$(POD_NAMESPACE)/nginx-load-balancer-microk8s-conf
    - --tcp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-tcp-microk8s-conf
    - --udp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-udp-microk8s-conf
    - --ingress-class=nginx
    - ' '
    - --publish-status-address=127.0.0.1
    `

after that, success. challenge completed and tls secrets get saved in the config. wonderfull. thanks for this project.

I'm glad you figured it out! These are my favorite kind of tickets 😊😊