Fatal 500 Crash On Startup
ProfMoo opened this issue · 6 comments
ProfMoo commented
Hi! Love this tool 😄, looking forward to simplifying my homelab with it.
I ran into an interesting error that I haven't been able to resolve. It happens consistently on startup.
time="2024-06-02T23:28:20Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s DefaultTargets:[] GlooNamespaces:[gloo-system] SkipperRouteGroupVersion:zalando.org/v1 Sources:[ingress service] Namespace: AnnotationFilter: LabelFilter: IngressClassNames:[] FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false IgnoreIngressRulesSpec:false GatewayNamespace: GatewayLabelFilter: Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:webhook GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s GoogleZoneVisibility: DomainFilter:[drmoo.io] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] TargetNetFilter:[] ExcludeTargetNets:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSAssumeRoleExternalID: AWSBatchChangeSize:1000 AWSBatchChangeSizeBytes:32000 AWSBatchChangeSizeValues:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AWSSDServiceCleanup:false AWSZoneMatchParent:false AWSDynamoDBRegion: AWSDynamoDBTable:external-dns AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatDNSConfiguration: BluecatConfigFile:/etc/kubernetes/bluecat.json BluecatDNSView: BluecatGatewayHost: BluecatRootZone: BluecatDNSServerName: BluecatDNSDeployType:no-deploy BluecatSkipTLSVerify:false CloudflareProxied:false CloudflareDNSRecordsPerPage:100 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 InfobloxFQDNRegEx: InfobloxNameRegEx: InfobloxCreatePTR:false InfobloxCacheDuration:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml OCICompartmentOCID: OCIAuthInstancePrincipal:false OCIZoneScope:GLOBAL OCIZoneCacheDuration:0s InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSSkipTLSVerify:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:main TXTPrefix:k8s.main. TXTSuffix: TXTEncryptEnabled:false TXTEncryptAESKey: Interval:1m0s MinEventSyncInterval:5s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:debug TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint: ExoscaleAPIKey: ExoscaleAPISecret: ExoscaleAPIEnvironment:api ExoscaleAPIZone:ch-gva-2 CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: ResolveServiceLoadBalancerHostname:false RFC2136Host: RFC2136Port:0 RFC2136Zone:[] RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s RFC2136BatchChangeSize:50 RFC2136UseTLS:false RFC2136SkipTLSVerify:false NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A AAAA CNAME] ExcludeDNSRecordTypes:[] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false OCPRouterName: IBMCloudProxied:false IBMCloudConfigFile:/etc/kubernetes/ibmcloud.json TencentCloudConfigFile:/etc/kubernetes/tencent-cloud.json TencentCloudZoneType: PiholeServer: PiholePassword: PiholeTLSInsecureSkipVerify:false PluralCluster: PluralProvider: WebhookProviderURL:http://localhost:8888 WebhookProviderReadTimeout:5s WebhookProviderWriteTimeout:10s WebhookServer:false TraefikDisableLegacy:false TraefikDisableNew:false}"
time="2024-06-02T23:28:20Z" level=info msg="Instantiating new Kubernetes client"
time="2024-06-02T23:28:20Z" level=debug msg="apiServerURL: "
time="2024-06-02T23:28:20Z" level=debug msg="kubeConfig: "
time="2024-06-02T23:28:20Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2024-06-02T23:28:20Z" level=info msg="Created Kubernetes client https://10.96.0.1:443"
time="2024-06-02T23:28:20Z" level=debug msg="Endpoints generated from ingress: cilium/hubble-ui: [hubble.drmoo.io 0 IN CNAME internal.drmoo.io []]"
time="2024-06-02T23:28:20Z" level=debug msg="Endpoints generated from ingress: media/prowlarr: [prowlarr.drmoo.io 0 IN CNAME internal.drmoo.io [] *.drmoo.io 0 IN CNAME internal.drmoo.io []]"
time="2024-06-02T23:28:20Z" level=debug msg="Endpoints generated from ingress: media/radarr: [radarr.internal.drmoo.io 0 IN CNAME internal.drmoo.io []]"
time="2024-06-02T23:28:20Z" level=debug msg="Endpoints generated from ingress: observability/grafana: [grafana.drmoo.io 0 IN CNAME internal.drmoo.io [] *.drmoo.io 0 IN CNAME internal.drmoo.io []]"
time="2024-06-02T23:28:20Z" level=debug msg="Endpoints generated from ingress: observability/kube-prometheus-stack-prometheus: [prometheus.drmoo.io 0 IN CNAME internal.drmoo.io [] *.drmoo.io 0 IN CNAME internal.drmoo.io []]"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service flux-system/webhook-receiver"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service kube-system/kube-prometheus-stack-kubelet"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service networking/nginx-external-controller-admission"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service networking/nginx-internal-controller-metrics"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service cert-manager/cert-manager-webhook"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/kube-prometheus-stack-prometheus"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service cilium/hubble-ui"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service kube-system/kube-prometheus-stack-kube-scheduler"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service networking/nginx-external-controller-metrics"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service cilium/cilium-agent"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service homelab/goldpinger"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/loki-gateway"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service flux-system/source-controller"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service media/sonarr-gui"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/kube-prometheus-stack-prometheus-node-exporter"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/prometheus-operated"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service cert-manager/cert-manager"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service cilium/hubble-peer"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service default/talos"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service kube-system/kube-prometheus-stack-coredns"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service media/radarr"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service cilium/cilium-operator"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/grafana"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/logging-operator-fluentd-headless"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/loki-memberlist"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service media/prowlarr"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/prometheus-gui"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/kube-state-metrics"
time="2024-06-02T23:28:20Z" level=debug msg="Endpoints generated from service: networking/nginx-internal-controller: [internal.drmoo.io 0 IN A 192.168.8.121 []]"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/kube-prometheus-stack-operator"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/logging-operator"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/proxmox"
time="2024-06-02T23:28:20Z" level=debug msg="Endpoints generated from service: networking/nginx-external-controller: [external.drmoo.io 0 IN A 192.168.8.126 []]"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service flux-system/notification-controller"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/loki-headless"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service cilium/hubble-relay"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service kube-system/kube-prometheus-stack-kube-etcd"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service networking/external-dns-unifi"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/logging-operator-fluentbit-monitor"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/logging-operator-fluentd"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service kube-system/kube-dns"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service cilium/hubble-metrics"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service media/sonarr"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service base/metrics-server"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service kube-system/host-dns"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service kube-system/kube-prometheus-stack-kube-controller-manager"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service media/prowlarr-gui"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/logging-operator-fluentd-metrics"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service homelab/homepage"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service media/radarr-gui"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service default/kubernetes"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service networking/nginx-internal-controller-admission"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/loki"
time="2024-06-02T23:28:20Z" level=debug msg="No endpoints could be generated from service observability/grafana-gui"
time="2024-06-02T23:28:20Z" level=debug msg="Removing duplicate endpoint *.drmoo.io 0 IN CNAME internal.drmoo.io []"
time="2024-06-02T23:28:20Z" level=debug msg="Removing duplicate endpoint *.drmoo.io 0 IN CNAME internal.drmoo.io []"
time="2024-06-02T23:28:20Z" level=debug msg="Failed to apply changes with code 500"
time="2024-06-02T23:28:20Z" level=fatal msg="Failed to do run once: failed to apply changes with code 500"
Version Tag: a3e54980abe00f9c511010952dcfd4dc15351cb474b337761cc4766d34292ca7
Here is my full configuration, in case that's helpful.
Seemingly removing duplicate endpoints is causing this problem, but I'm not sure.
time="2024-06-02T23:28:20Z" level=debug msg="Removing duplicate endpoint *.drmoo.io 0 IN CNAME internal.drmoo.io []"
Please let me know if there is additional information which might be helpful here.
onedr0p commented
This appears to be an issue with using a wildcard DNS, can you explain how you are defining your ingress annotations and where the wildcard DNS record is coming from? You shouldn't need to use a wildcard DNS record.
onedr0p commented
I think I see where the issue is, your TLS blocks are getting read as well by external-dns ...
You can probably remove these TLS blocks, or in the external-dns helm values add this arg ...
ProfMoo commented
Any luck @ProfMoo ?
Thanks for the suggestions 😄. Things suddenly started working again earlier this morning (i.e. no crashing). I haven't been able to pin down what changed yet. Either way I'll probably add that external-dns value you suggested, seems to be the right move for my setup.
I'll followup if I find anything more definitive, but feel free to close this issue 👍
bdsoha commented
I am having the same issue with traefik.io/v1alpha1 and IngressRoutes without wildcard definitions.