CoreDNS issue with minikube when using podman rootless driver

Question

CoreDNS issue with minikube when using podman rootless driver

loungerider opened this issue a month ago · 2 comments

What Happened?

Tested on macOS Sonoma 14.4.1 Darwin Kernel Version 23.4.0 x86_64

minikube start --addons=ingress --driver=podman --container-runtime=containerd

minikube v1.33.1 on Darwin 14.4.1

Followed the directions at https://kubernetes.io/docs/tasks/access-application-cluster/ingress-minikube/ and it works with podman in rootful mode. When using the podman driver in rootless mode accessing the ingress through minikube tunnel will timeout. It seems that DNS in the ingress-nginx-controller pod is not working which causes nginx to return a 504 timeout.

k exec -it pod/ingress-nginx-controller-768f948f8f-dgj8g -n ingress-nginx -- /bin/bash
ingress-nginx-controller-768f948f8f-dgj8g:/etc/nginx$ cat /etc/resolv.conf
search ingress-nginx.svc.cluster.local svc.cluster.local cluster.local dns.podman
nameserver 10.96.0.10
options ndots:5
ingress-nginx-controller-768f948f8f-dgj8g:/etc/nginx$ nslookup kubernetes.default.svc.cluster.local
;; connection timed out; no servers could be reached

Also tested using a busybox pod and found the same DNS issue.

CoreDNS logs when running in rootless mode

k logs coredns-7db6d8ff4d-wnq8g -n kube-system
.:53
[INFO] plugin/reload: Running configuration SHA512 = 0acd057f3a0f4709031c7dfc71869eb076b357e33cc3f9e8c7bbf24d03af38ef7635b34367a89d45adab17a5391a1c2d058603c581e1c5f4a21732bf72371934
CoreDNS-1.11.1
linux/amd64, go1.20.7, ae2bbc2
[INFO] 127.0.0.1:57034 - 5798 "HINFO IN 5078784625045588880.5657588632774814912. udp 57 false 512" - - 0 6.003962814s
[ERROR] plugin/errors: 2 5078784625045588880.5657588632774814912. HINFO: read udp 10.244.0.2:50533->192.168.49.1:53: i/o timeout
[INFO] 127.0.0.1:55865 - 43166 "HINFO IN 5078784625045588880.5657588632774814912. udp 57 false 512" - - 0 6.002597212s
[ERROR] plugin/errors: 2 5078784625045588880.5657588632774814912. HINFO: read udp 10.244.0.2:36520->192.168.49.1:53: i/o timeout
[INFO] 127.0.0.1:58431 - 18170 "HINFO IN 5078784625045588880.5657588632774814912. udp 57 false 512" - - 0 4.00176304s
[ERROR] plugin/errors: 2 5078784625045588880.5657588632774814912. HINFO: read udp 10.244.0.2:37051->192.168.49.1:53: i/o timeout
[INFO] 127.0.0.1:38790 - 57209 "HINFO IN 5078784625045588880.5657588632774814912. udp 57 false 512" - - 0 2.001476848s
[ERROR] plugin/errors: 2 5078784625045588880.5657588632774814912. HINFO: read udp 10.244.0.2:52216->192.168.49.1:53: i/o timeout
[INFO] 127.0.0.1:40645 - 34316 "HINFO IN 5078784625045588880.5657588632774814912. udp 57 false 512" - - 0 2.000348752s
[ERROR] plugin/errors: 2 5078784625045588880.5657588632774814912. HINFO: read udp 10.244.0.2:36570->192.168.49.1:53: i/o timeout
[INFO] 127.0.0.1:53601 - 57548 "HINFO IN 5078784625045588880.5657588632774814912. udp 57 false 512" - - 0 2.000684865s

CoreDNS logs when running in rootful mode - working

 k logs coredns-7db6d8ff4d-b5xkh -n kube-system
.:53
[INFO] plugin/reload: Running configuration SHA512 = 591cf328cccc12bc490481273e738df59329c62c0b729d94e8b61db9961c2fa5f046dd37f1cf888b953814040d180f52594972691cd6ff41be96639138a43908
CoreDNS-1.11.1
linux/amd64, go1.20.7, ae2bbc2

Attach the log file

log.txt

Operating System

macOS (Default)

Driver

Podman