kubernetes/examples

Getting DNS errors in example guestbook-go, involving failure to resolve "redis_slave"

dkwgit opened this issue · 0 comments

dkwgit commented

UPDATE. I completely missed Antoine Toussaint's prior issue 506, AntoineToussaint.
He diagnosed the same things. His proposed fix is the better one, however, my one line fix to the DNS issue, see #5 below, gets things going quickly. Like he found, I also did also need to make a slight adjustment to the go import to correctly import simpleredis. It needs to be imported as: github.com/xyproto/simpleredis/v2

Original Issue:
It's possible this issue is a bug in the guestbook-go example. Or, maybe it is just a problem with Docker Desktop K8s, which is what i was using. I don't know. I'm creating the issue to possibly help others who run into the same problem.

  1. The go program in guestbook-go/main.go shows this panic after adding list items in the guestbook (and the items never show in the web app):

    PANIC: dial tcp: lookup redis-slave on 10.96.0.10:53: server misbehaving
    goroutine 63 [running]:
    github.com/codegangsta/negroni.(*Recovery).ServeHTTP.func1(0x7f954fac7b58, 0xc82007f100, 0xc8200ecd20)
        /go/src/github.com/codegangsta/negroni/recovery.go:34 +0xe9
    panic(0x7a8c60, 0xc820122f00)
        /usr/local/go/src/runtime/panic.go:426 +0x4e9
    main.HandleError(0x6e2780, 0xc8201a5980, 0x7f954fa7e0a8, 0xc820122f00, 0x0, 0x0)
        /go/src/github.com/GoogleCloudPlatform/kubernetes/examples/guestbook-go/_src/main.go:71 +0x59
    main.ListRangeHandler(0x7f954fac7b58, 0xc82007f100, 0xc8200f87e0)
    
  2. Using dnsutils via kubectl exec -i -t dnsutils -- nslookup redis-slave shows:

    Server:         10.96.0.10
    Address:        10.96.0.10#53
    
    ** server can't find redis-slave: SERVFAIL
    
    command terminated with exit code 1
    
  3. Logs from kube-dns, via kubectl logs --namespace=kube-system -l k8s-app=kube-dns show:

    [INFO] 10.1.0.67:35518 - 32059 "A IN redis-slave.svc.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 140 0.000136801s
    [INFO] 10.1.0.67:48817 - 11700 "AAAA IN redis-slave.cluster.local. udp 43 false 512" NXDOMAIN qr,aa,rd 136 0.0000953s
    [INFO] 10.1.0.67:34640 - 708 "A IN redis-slave.cluster.local. udp 43 false 512" NXDOMAIN qr,aa,rd 136 0.0000894s
    [INFO] 10.1.0.67:33927 - 62058 "A IN redis-slave. udp 29 false 512" - - 0 2.001422668s
    [ERROR] plugin/errors: 2 redis-slave. A: read udp 10.1.0.50:44112->192.168.65.7:53: i/o timeout
    
  4. However, redis-master (from redis-master-service.yaml), does resolve (i.e. DNS is working, in some fashion), per kubectl exec -i -t dnsutils -- nslookup redis-master

    Server:         10.96.0.10
    Address:        10.96.0.10#53
    
    Name:   redis-master.default.svc.cluster.local
    Address: 10.109.231.172
    
  5. A one line change fixes this for me. (Not saying this is the right thing to do for the whole example for everyone--I'd have to research more--but it might help other Docker Desktop K8S users to get the example working, if they are experiencing the same problem).

    If I change the service name in redis-replica-service.yaml on line 4 to
    name: redis-slave

    the web app starts working (displaying added items) and the DNS errors stop.

  6. Additional Info on versions/systems I used and changes I made to the example. (I have not tried this with a K8S other than Docker Desktop's built-in).

  • System versions

    Docker Desktop 4.24.1 (123237)
    Docker Engine v24.0.6
    Kubernetes (Bundled with Docker Desktop) 1.27.2
    Windows 11 system (However I was developing inside a Debian derived dev container hosted in Docker with a WSL 2 backend.

  • Code changes

    This version of Docker Engine is always using docker buildx when docker build is invoked, so I changed guestbook-go/Makefile to use 'docker build', not 'docker buildx'
    The example is fairly stale--to get it to build, I changed the import of simpleredis in guestbook-go/main.go to use v2 (following is the changed line within import list): simpleredis "github.com/xyproto/simpleredis/v2". I made the same change (adding '/v2') at line 18 in "guestbook-go/Makefile"