nabsul/kcert

Certificate renewal failed

koehoorn opened this issue · 3 comments

koehoorn commented

Hi,
i dont know how to debug this further.
my setup is arm64 ( pine64 boards) running ubuntu and microk8s with the metallb software load balancer leading into ingress-nginx controller. then i created an nginx 'hello world' pod with persistent storage and a service and an ingress, so far works well on http.

Now i want to add tls termination on the ingress controller and found your project. this looks much cleaner then cert-manager.
so i defined the label and the tls section in the ingress definition and i see kcert responding. it tries to create secrets and get it signed, but that last part seems to fail. ''renewal failed", as you can see from the error logs below.

  • tried production url too.
  • tried longer ACME__VALIDATIONWAITTIMESECONDS 60

as soon as i define a tls section in the ingress, i get a certificate from ca 'Kubernetes Ingress Controller Fake Certificate' and it redirects http into https. i cant see my hello world message on http anymore. (perhaps thats why the challenge is unaccessible?)
i am guessing it is unable to host the challenge token on http. How does this part work and how to trace and debug this?

kubectl logs -n kcert kcert-56c78dc7cc-949hq
{"EventId":0,"LogLevel":"Information","Category":"KCert.Services.RenewalHandler","Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","State":{"Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","authUri":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283","status":"invalid","{OriginalFormat}":"Get Auth {authUri}: {status}"}} {"EventId":0,"LogLevel":"Information","Category":"KCert.Services.RenewalHandler","Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","State":{"Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","authUri":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283","status":"invalid","{OriginalFormat}":"Get Auth {authUri}: {status}"}} {"EventId":0,"LogLevel":"Information","Category":"KCert.Services.RenewalHandler","Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","State":{"Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","authUri":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283","status":"invalid","{OriginalFormat}":"Get Auth {authUri}: {status}"}} {"EventId":0,"LogLevel":"Information","Category":"KCert.Services.RenewalHandler","Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","State":{"Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","authUri":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283","status":"invalid","{OriginalFormat}":"Get Auth {authUri}: {status}"}} {"EventId":0,"LogLevel":"Information","Category":"KCert.Services.RenewalHandler","Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","State":{"Message":"Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283: invalid","authUri":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283","status":"invalid","{OriginalFormat}":"Get Auth {authUri}: {status}"}} {"EventId":0,"LogLevel":"Error","Category":"KCert.Services.RenewalHandler","Message":"Certificate renewal failed.","Exception":"System.Exception: Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283 did not complete in time. Last Response: invalid at KCert.Services.RenewalHandler.ValidateAuthorizationAsync(String key, String kid, String nonce, Uri authUri) in /build/Services/RenewalHandler.cs:line 111 at KCert.Services.RenewalHandler.RenewCertAsync(String ns, String secretName, String[] hosts) in /build/Services/RenewalHandler.cs:line 45","State":{"Message":"Certificate renewal failed.","{OriginalFormat}":"Certificate renewal failed."}} {"EventId":0,"LogLevel":"Error","Category":"KCert.Services.KCertClient","Message":"Renewal failed","Exception":"KCert.Models.RenewalException: Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283 did not complete in time. Last Response: invalid ---\u003E System.Exception: Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3543724283 did not complete in time. Last Response: invalid at KCert.Services.RenewalHandler.ValidateAuthorizationAsync(String key, String kid, String nonce, Uri authUri) in /build/Services/RenewalHandler.cs:line 111 at KCert.Services.RenewalHandler.RenewCertAsync(String ns, String secretName, String[] hosts) in /build/Services/RenewalHandler.cs:line 45 --- End of inner exception stack trace --- at KCert.Services.RenewalHandler.RenewCertAsync(String ns, String secretName, String[] hosts) in /build/Services/RenewalHandler.cs:line 57 at KCert.Services.KCertClient.RenewCertAsync(Task prev, String ns, String secretName, String[] hosts, CancellationToken tok) in /build/Services/KCertClient.cs:line 63","State":{"Message":"Renewal failed","{OriginalFormat}":"Renewal failed"}} {"EventId":0,"LogLevel":"Information","Category":"KCert.Services.EmailClient","Message":"Cannot send email email because it\u0027s not configured correctly","State":{"Message":"Cannot send email email because it\u0027s not configured correctly","{OriginalFormat}":"Cannot send email email because it\u0027s not configured correctly"}}

koehoorn commented

microk8s uses 'ingress.class public', found via
https://stackoverflow.com/questions/54506269/simple-ingress-from-host-with-microk8s
so i tried to set kubernetes.io/ingress.class: public in the ingress definition, but same effect.

koehoorn commented

okay i rubberducked this aperantly. ;)

i assumed you used/assumed the ingress.class annotation to be nginx in your code. so i changed the ingress controller daemonset to define nginx.

kubectl edit ds nginx-ingress-microk8s-controller -n ingress

`

  • args:
    - /nginx-ingress-controller
    - --configmap=$(POD_NAMESPACE)/nginx-load-balancer-microk8s-conf
    - --tcp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-tcp-microk8s-conf
    - --udp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-udp-microk8s-conf
    - --ingress-class=nginx
    - ' '
    - --publish-status-address=127.0.0.1
    `

after that, success. challenge completed and tls secrets get saved in the config. wonderfull. thanks for this project.

  • ❤️1
nabsul commented

I'm glad you figured it out! These are my favorite kind of tickets 😊😊