Renewal failed using ingress-nginx
Closed this issue · 2 comments
AlexanderArvidsson commented
Hi,
I can't for the life of me figure out why the certificate won't validate and get saved as a secret.
I'm on Azure, using the Ingress-NGINX Controller, which I only just installed through the AKS guide and I am able to access my pods without TLS, if I set up an ingress for it. When I'm setting up KCert with TLS, I get errors about failed renewals.
I had KCert set up on a previous cluster and it worked flawlessly, but now, using the exact same configurations as last time, it won't work.
My DNS is set up and I have verified that it has propagated.
Here's my ingress and logs:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
labels:
kcert.dev/ingress: "managed"
spec:
ingressClassName: nginx
tls:
- hosts:
- stage.kla.sh
secretName: stage-tls
rules:
- host: stage.kla.sh
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: app-api
port:
number: 80
Here's the logs from svc/kcert:
{
"Timestamp": "2023:10:02 02:41:52.255820",
"EventId": 60,
"LogLevel": "Warning",
"Category": "Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository",
"Message": "Storing keys in a directory \u0027/root/.aspnet/DataProtection-Keys\u0027 that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.",
"State": {
"Message": "Storing keys in a directory \u0027/root/.aspnet/DataProtection-Keys\u0027 that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.",
"path": "/root/.aspnet/DataProtection-Keys",
"{OriginalFormat}": "Storing keys in a directory \u0027{path}\u0027 that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed."
}
}
{
"Timestamp": "2023:10:02 02:41:52.261320",
"EventId": 62,
"LogLevel": "Information",
"Category": "Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager",
"Message": "User profile is available. Using \u0027/root/.aspnet/DataProtection-Keys\u0027 as key repository; keys will not be encrypted at rest.",
"State": {
"Message": "User profile is available. Using \u0027/root/.aspnet/DataProtection-Keys\u0027 as key repository; keys will not be encrypted at rest.",
"FullName": "/root/.aspnet/DataProtection-Keys",
"{OriginalFormat}": "User profile is available. Using \u0027{FullName}\u0027 as key repository; keys will not be encrypted at rest."
}
}
{
"Timestamp": "2023:10:02 02:41:52.307266",
"EventId": 58,
"LogLevel": "Information",
"Category": "Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager",
"Message": "Creating key {f05a6c1a-a8e4-40c4-8971-23a7d31bdd7a} with creation date 2023-10-02 14:41:52Z, activation date 2023-10-02 14:41:52Z, and expiration date 2023-12-31 14:41:52Z.",
"State": {
"Message": "Creating key {f05a6c1a-a8e4-40c4-8971-23a7d31bdd7a} with creation date 2023-10-02 14:41:52Z, activation date 2023-10-02 14:41:52Z, and expiration date 2023-12-31 14:41:52Z.",
"KeyId": "f05a6c1a-a8e4-40c4-8971-23a7d31bdd7a",
"CreationDate": "10/02/2023 14:41:52 \u002B00:00",
"ActivationDate": "10/02/2023 14:41:52 \u002B00:00",
"ExpirationDate": "12/31/2023 14:41:52 \u002B00:00",
"{OriginalFormat}": "Creating key {KeyId:B} with creation date {CreationDate:u}, activation date {ActivationDate:u}, and expiration date {ExpirationDate:u}."
}
}
{
"Timestamp": "2023:10:02 02:41:52.314481",
"EventId": 35,
"LogLevel": "Warning",
"Category": "Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager",
"Message": "No XML encryptor configured. Key {f05a6c1a-a8e4-40c4-8971-23a7d31bdd7a} may be persisted to storage in unencrypted form.",
"State": {
"Message": "No XML encryptor configured. Key {f05a6c1a-a8e4-40c4-8971-23a7d31bdd7a} may be persisted to storage in unencrypted form.",
"KeyId": "f05a6c1a-a8e4-40c4-8971-23a7d31bdd7a",
"{OriginalFormat}": "No XML encryptor configured. Key {KeyId:B} may be persisted to storage in unencrypted form."
}
}
{
"Timestamp": "2023:10:02 02:41:52.318166",
"EventId": 39,
"LogLevel": "Information",
"Category": "Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository",
"Message": "Writing data to file \u0027/root/.aspnet/DataProtection-Keys/key-f05a6c1a-a8e4-40c4-8971-23a7d31bdd7a.xml\u0027.",
"State": {
"Message": "Writing data to file \u0027/root/.aspnet/DataProtection-Keys/key-f05a6c1a-a8e4-40c4-8971-23a7d31bdd7a.xml\u0027.",
"FileName": "/root/.aspnet/DataProtection-Keys/key-f05a6c1a-a8e4-40c4-8971-23a7d31bdd7a.xml",
"{OriginalFormat}": "Writing data to file \u0027{FileName}\u0027."
}
}
{
"Timestamp": "2023:10:02 02:41:52.332427",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalService",
"Message": "Starting up renewal service.",
"State": {
"Message": "Starting up renewal service.",
"{OriginalFormat}": "Starting up renewal service."
}
}
{
"Timestamp": "2023:10:02 02:41:52.337300",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalService",
"Message": "Checking for certs that need renewals...",
"State": {
"Message": "Checking for certs that need renewals...",
"{OriginalFormat}": "Checking for certs that need renewals..."
}
}
{
"Timestamp": "2023:10:02 02:41:52.355889",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.IngressMonitorService",
"Message": "Watching for ingress is enabled",
"State": {
"Message": "Watching for ingress is enabled",
"{OriginalFormat}": "Watching for ingress is enabled"
}
}
{
"Timestamp": "2023:10:02 02:41:52.361210",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.IngressMonitorService",
"Message": "Watching for ingress changes",
"State": {
"Message": "Watching for ingress changes",
"{OriginalFormat}": "Watching for ingress changes"
}
}
{
"Timestamp": "2023:10:02 02:41:52.366199",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.K8sClient",
"Message": "Starting watch request for V1Ingress[kcert.dev/ingress=managed]",
"State": {
"Message": "Starting watch request for V1Ingress[kcert.dev/ingress=managed]",
"type": "V1Ingress",
"label": "kcert.dev/ingress=managed",
"{OriginalFormat}": "Starting watch request for {type}[{label}]"
}
}
{
"Timestamp": "2023:10:02 02:41:52.379260",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.ConfigMonitorService",
"Message": "Watching for configmaps is enabled",
"State": {
"Message": "Watching for configmaps is enabled",
"{OriginalFormat}": "Watching for configmaps is enabled"
}
}
{
"Timestamp": "2023:10:02 02:41:52.379920",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.ConfigMonitorService",
"Message": "Watching for configmaps changes",
"State": {
"Message": "Watching for configmaps changes",
"{OriginalFormat}": "Watching for configmaps changes"
}
}
{
"Timestamp": "2023:10:02 02:41:52.380545",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.K8sClient",
"Message": "Starting watch request for V1ConfigMap[kcert.dev/cert-request=request]",
"State": {
"Message": "Starting watch request for V1ConfigMap[kcert.dev/cert-request=request]",
"type": "V1ConfigMap",
"label": "kcert.dev/cert-request=request",
"{OriginalFormat}": "Starting watch request for {type}[{label}]"
}
}
{
"Timestamp": "2023:10:02 02:41:52.389136",
"EventId": 0,
"LogLevel": "Warning",
"Category": "Microsoft.AspNetCore.Server.Kestrel",
"Message": "Overriding address(es) \u0027http://\u002B:80\u0027. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.",
"State": {
"Message": "Overriding address(es) \u0027http://\u002B:80\u0027. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.",
"addresses": "http://\u002B:80",
"{OriginalFormat}": "Overriding address(es) \u0027{addresses}\u0027. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead."
}
}
{
"Timestamp": "2023:10:02 02:41:52.393837",
"EventId": 14,
"LogLevel": "Information",
"Category": "Microsoft.Hosting.Lifetime",
"Message": "Now listening on: http://[::]:80",
"State": {
"Message": "Now listening on: http://[::]:80",
"address": "http://[::]:80",
"{OriginalFormat}": "Now listening on: {address}"
}
}
{
"Timestamp": "2023:10:02 02:41:52.393874",
"EventId": 14,
"LogLevel": "Information",
"Category": "Microsoft.Hosting.Lifetime",
"Message": "Now listening on: http://[::]:8080",
"State": {
"Message": "Now listening on: http://[::]:8080",
"address": "http://[::]:8080",
"{OriginalFormat}": "Now listening on: {address}"
}
}
{
"Timestamp": "2023:10:02 02:41:52.393962",
"EventId": 0,
"LogLevel": "Information",
"Category": "Microsoft.Hosting.Lifetime",
"Message": "Application started. Press Ctrl\u002BC to shut down.",
"State": {
"Message": "Application started. Press Ctrl\u002BC to shut down.",
"{OriginalFormat}": "Application started. Press Ctrl\u002BC to shut down."
}
}
{
"Timestamp": "2023:10:02 02:41:52.393997",
"EventId": 0,
"LogLevel": "Information",
"Category": "Microsoft.Hosting.Lifetime",
"Message": "Hosting environment: Production",
"State": {
"Message": "Hosting environment: Production",
"envName": "Production",
"{OriginalFormat}": "Hosting environment: {envName}"
}
}
{
"Timestamp": "2023:10:02 02:41:52.394018",
"EventId": 0,
"LogLevel": "Information",
"Category": "Microsoft.Hosting.Lifetime",
"Message": "Content root path: /app/",
"State": {
"Message": "Content root path: /app/",
"contentRoot": "/app/",
"{OriginalFormat}": "Content root path: {contentRoot}"
}
}
{
"Timestamp": "2023:10:02 02:41:52.547276",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalService",
"Message": "Renewal check completed.",
"State": {
"Message": "Renewal check completed.",
"{OriginalFormat}": "Renewal check completed."
}
}
{
"Timestamp": "2023:10:02 02:41:52.547804",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalService",
"Message": "Sleeping for 06:00:00",
"State": {
"Message": "Sleeping for 06:00:00",
"renewalTime": "06:00:00",
"{OriginalFormat}": "Sleeping for {renewalTime}"
}
}
{
"Timestamp": "2023:10:02 02:41:52.568214",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.IngressMonitorService",
"Message": "Ingress change event [Added] for app-app-ingress",
"State": {
"Message": "Ingress change event [Added] for app-app-ingress",
"type": "Added",
"ns": "app",
"name": "app-ingress",
"{OriginalFormat}": "Ingress change event [{type}] for {ns}-{name}"
}
}
{
"Timestamp": "2023:10:02 02:41:52.572604",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.CertChangeService",
"Message": "Waiting for semaphore",
"State": {
"Message": "Waiting for semaphore",
"{OriginalFormat}": "Waiting for semaphore"
}
}
{
"Timestamp": "2023:10:02 02:41:52.572634",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.CertChangeService",
"Message": "Starting check for changes.",
"State": {
"Message": "Starting check for changes.",
"{OriginalFormat}": "Starting check for changes."
}
}
{
"Timestamp": "2023:10:02 02:41:52.587005",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.CertChangeService",
"Message": "Processing ingress app:app-ingress",
"State": {
"Message": "Processing ingress app:app-ingress",
"ns": "app",
"n": "app-ingress",
"{OriginalFormat}": "Processing ingress {ns}:{n}"
}
}
{
"Timestamp": "2023:10:02 02:41:52.587066",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.CertChangeService",
"Message": "Processing secret stage-tls",
"State": {
"Message": "Processing secret stage-tls",
"s": "stage-tls",
"{OriginalFormat}": "Processing secret {s}"
}
}
{
"Timestamp": "2023:10:02 02:41:52.604115",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.CertChangeService",
"Message": "Handling cert app - stage-tls hosts: stage.kla.sh",
"State": {
"Message": "Handling cert app - stage-tls hosts: stage.kla.sh",
"ns": "app",
"name": "stage-tls",
"h": "stage.kla.sh",
"{OriginalFormat}": "Handling cert {ns} - {name} hosts: {h}"
}
}
{
"Timestamp": "2023:10:02 02:41:52.683335",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.KCertClient",
"Message": "Giving challenge ingress time to propagate",
"State": {
"Message": "Giving challenge ingress time to propagate",
"{OriginalFormat}": "Giving challenge ingress time to propagate"
}
}
{
"Timestamp": "2023:10:02 02:41:58.877374",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalHandler",
"Message": "Initialized renewal process for secret app/stage-tls - hosts stage.kla.sh - kid https://acme-staging-v02.api.letsencrypt.org/acme/acct/120470404",
"State": {
"Message": "Initialized renewal process for secret app/stage-tls - hosts stage.kla.sh - kid https://acme-staging-v02.api.letsencrypt.org/acme/acct/120470404",
"ns": "app",
"secretName": "stage-tls",
"hosts": "stage.kla.sh",
"kid": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/120470404",
"{OriginalFormat}": "Initialized renewal process for secret {ns}/{secretName} - hosts {hosts} - kid {kid}"
}
}
{
"Timestamp": "2023:10:02 02:41:59.096077",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalHandler",
"Message": "Created order: pending",
"State": {
"Message": "Created order: pending",
"status": "pending",
"{OriginalFormat}": "Created order: {status}"
}
}
{
"Timestamp": "2023:10:02 02:41:59.096808",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalHandler",
"Message": "Order https://acme-staging-v02.api.letsencrypt.org/acme/order/120470404/11324245514 created with finalizeUri https://acme-staging-v02.api.letsencrypt.org/acme/finalize/120470404/11324245514",
"State": {
"Message": "Order https://acme-staging-v02.api.letsencrypt.org/acme/order/120470404/11324245514 created with finalizeUri https://acme-staging-v02.api.letsencrypt.org/acme/finalize/120470404/11324245514",
"orderUri": "https://acme-staging-v02.api.letsencrypt.org/acme/order/120470404/11324245514",
"finalizeUri": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/120470404/11324245514",
"{OriginalFormat}": "Order {orderUri} created with finalizeUri {finalizeUri}"
}
}
{
"Timestamp": "2023:10:02 02:41:59.276130",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalHandler",
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: pending",
"State": {
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: pending",
"authUri": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444",
"status": "pending",
"{OriginalFormat}": "Get Auth {authUri}: {status}"
}
}
{
"Timestamp": "2023:10:02 02:41:59.450700",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalHandler",
"Message": "TriggerChallenge https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/8634101444/hTMMqw: pending",
"State": {
"Message": "TriggerChallenge https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/8634101444/hTMMqw: pending",
"challengeUri": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/8634101444/hTMMqw",
"status": "pending",
"{OriginalFormat}": "TriggerChallenge {challengeUri}: {status}"
}
}
{
"Timestamp": "2023:10:02 02:42:09.623043",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalHandler",
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: pending",
"State": {
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: pending",
"authUri": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444",
"status": "pending",
"{OriginalFormat}": "Get Auth {authUri}: {status}"
}
}
{
"Timestamp": "2023:10:02 02:42:19.798515",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalHandler",
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: invalid",
"State": {
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: invalid",
"authUri": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444",
"status": "invalid",
"{OriginalFormat}": "Get Auth {authUri}: {status}"
}
}
{
"Timestamp": "2023:10:02 02:42:29.969204",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalHandler",
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: invalid",
"State": {
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: invalid",
"authUri": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444",
"status": "invalid",
"{OriginalFormat}": "Get Auth {authUri}: {status}"
}
}
{
"Timestamp": "2023:10:02 02:42:40.136361",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalHandler",
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: invalid",
"State": {
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: invalid",
"authUri": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444",
"status": "invalid",
"{OriginalFormat}": "Get Auth {authUri}: {status}"
}
}
{
"Timestamp": "2023:10:02 02:42:50.313571",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalHandler",
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: invalid",
"State": {
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: invalid",
"authUri": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444",
"status": "invalid",
"{OriginalFormat}": "Get Auth {authUri}: {status}"
}
}
{
"Timestamp": "2023:10:02 02:43:00.486866",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.RenewalHandler",
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: invalid",
"State": {
"Message": "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: invalid",
"authUri": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444",
"status": "invalid",
"{OriginalFormat}": "Get Auth {authUri}: {status}"
}
}
{
"Timestamp": "2023:10:02 02:43:00.487232",
"EventId": 0,
"LogLevel": "Error",
"Category": "KCert.Services.RenewalHandler",
"Message": "Certificate renewal failed.",
"Exception": "System.Exception: Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444 did not complete in time. Last Response: invalid\n at KCert.Services.RenewalHandler.ValidateAuthorizationAsync(String key, String kid, String nonce, Uri authUri) in /build/Services/RenewalHandler.cs:line 105\n at KCert.Services.RenewalHandler.RenewCertAsync(String ns, String secretName, String[] hosts) in /build/Services/RenewalHandler.cs:line 44",
"State": {
"Message": "Certificate renewal failed.",
"{OriginalFormat}": "Certificate renewal failed."
}
}
{
"Timestamp": "2023:10:02 02:43:00.497070",
"EventId": 0,
"LogLevel": "Error",
"Category": "KCert.Services.KCertClient",
"Message": "Renewal failed",
"Exception": "KCert.Models.RenewalException: Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444 did not complete in time. Last Response: invalid\n ---\u003E System.Exception: Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444 did not complete in time. Last Response: invalid\n at KCert.Services.RenewalHandler.ValidateAuthorizationAsync(String key, String kid, String nonce, Uri authUri) in /build/Services/RenewalHandler.cs:line 105\n at KCert.Services.RenewalHandler.RenewCertAsync(String ns, String secretName, String[] hosts) in /build/Services/RenewalHandler.cs:line 44\n --- End of inner exception stack trace ---\n at KCert.Services.RenewalHandler.RenewCertAsync(String ns, String secretName, String[] hosts) in /build/Services/RenewalHandler.cs:line 56\n at KCert.Services.KCertClient.RenewCertAsync(Task prev, String ns, String secretName, String[] hosts, CancellationToken tok) in /build/Services/KCertClient.cs:line 84",
"State": {
"Message": "Renewal failed",
"{OriginalFormat}": "Renewal failed"
}
}
{
"Timestamp": "2023:10:02 02:43:00.499841",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.EmailClient",
"Message": "Cannot send email email because it\u0027s not configured correctly",
"State": {
"Message": "Cannot send email email because it\u0027s not configured correctly",
"{OriginalFormat}": "Cannot send email email because it\u0027s not configured correctly"
}
}
{
"Timestamp": "2023:10:02 02:43:00.499962",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.CertChangeService",
"Message": "Check for changes completed.",
"State": {
"Message": "Check for changes completed.",
"{OriginalFormat}": "Check for changes completed."
}
}
I believe the most noteworthy message is "Get Auth https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/8634101444: invalid". I take it as the message from Let's Encrypt is received by KCert, but it's an invalid format?
Could you give me some guidance how to debug this further, what could be wrong, and what I can do to fix it?
Thanks!
AlexanderArvidsson commented
Not sure what I did different, I deleted my full namespace (not the kcert one, my actual application namespace), I set it up from scratch according to the internal documentation I am writing, and this time it worked flawlessly like I'm used to.
The only difference is I used a different name for the namespace, but other than that it was completely the same.
I suspect ingress-nginx had trouble working within that namespace, perhaps it entered an invalid state because I had re-installed it several times to try and fix this problem.
The steps I did to ensure everything worked, in case anyone else is interested:
- Create an empty namespace
- Set up ingress-nginx using the AKS Guide, I used the customized config because we use our own ACR.
- Set up the Azure demo voting app, this is only to test the load balancer works as it should:
apiVersion: apps/v1
kind: Deployment
metadata:
name: azure-vote-back
spec:
replicas: 1
selector:
matchLabels:
app: azure-vote-back
template:
metadata:
labels:
app: azure-vote-back
spec:
nodeSelector:
"kubernetes.io/os": linux
containers:
- name: azure-vote-back
image: mcr.microsoft.com/oss/bitnami/redis:6.0.8
env:
- name: ALLOW_EMPTY_PASSWORD
value: "yes"
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 250m
memory: 256Mi
ports:
- containerPort: 6379
name: redis
---
apiVersion: v1
kind: Service
metadata:
name: azure-vote-back
spec:
ports:
- port: 6379
selector:
app: azure-vote-back
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: azure-vote-front
spec:
replicas: 1
selector:
matchLabels:
app: azure-vote-front
template:
metadata:
labels:
app: azure-vote-front
spec:
nodeSelector:
"kubernetes.io/os": linux
containers:
- name: azure-vote-front
image: mcr.microsoft.com/azuredocs/azure-vote-front:v1
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 250m
memory: 256Mi
ports:
- containerPort: 80
env:
- name: REDIS
value: "azure-vote-back"
---
apiVersion: v1
kind: Service
metadata:
name: azure-vote-front
spec:
ports:
- port: 80
selector:
app: azure-vote-front- Set up a basic ingress with no TLS and no KCert:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-world-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: azure-vote-front
port:
number: 80- See if you can access the app through the external IP of the load balancer. If you can, great, continue down.
- Install KCert with Staging.
- Add TLS options to the ingress (check KCert documentation)
- Make sure to monitor
svc/kcertlogs. - Check if your requests come back as valid.
- If you get
invalid, I'm pretty sure it's your load balancer that is incorrectly set up. - Finally, delete the voting app and (optionally) the ingress. You can keep the ingress and just re-point it to another service.
Thanks for the great library, I hope no one runs into this problem I had initially since it was very tedious to debug!
nabsul commented
Hi! Thanks for sharing your experience! I'm glad you were able to get past the issue.
It does indeed sound like there was some kind of routing problem. Based on the first set of logs you shared, it seems like Let's Encrypt was not able to reach KCert to do the http validation.