canonical/microcloud

Add option to disable ipv6 protocol in microcloud init command.

Closed this issue · 4 comments

tregubovav-dev commented

Problem description:

microcloud init require IPv4 and IPv6 stack.
microcloud use calls mdns.DefaultParams(service) which instructs to use ipv4 and ipv6 sockets in msdn Query function call.
microcloud init fails with message Error: Failed lookup: write udp6 [::]:<SRC_PORT>->[ff02::fb]:5353: sendto: cannot assign requested address in case if ipv6 is disabled.

Prerequisites

  • The single instance with the single network interface is enough for reproduction
  • install microcloud via snap: snap install microcloud (neverminded which channel is used, all are impacted)
  • disable ipv6 stack (for example using shell command: sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1; sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1)

Steps to reproduce:

  • run sudo microcloud init --auto
    execution stops after Scanning for eligible servers ... step with the message Error: Failed lookup: write udp6 [::]:<SRC_PORT>->[ff02::fb]:5353: sendto: cannot assign requested address

Recommendations to fix
Add option to command disable ipv6 protocol in microcloud init command.

roosterfish commented

Hi @tregubovav-dev, thanks for reporting this. When disabling IPv6 using sysctl this is causing a panic on latest/edge:

root@mc:~# sysctl -w net.ipv6.conf.all.disable_ipv6=1; sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
root@mc:~# microcloud init --auto
Waiting for LXD to start...
Using address "10.146.93.183" for MicroCloud
Scanning for eligible servers ...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5d77ae]

goroutine 1 [running]:
net.(*UDPConn).SyscallConn(0xc0001406c0?)
	net/udpsock.go:129 +0xe
golang.org/x/net/internal/socket.NewConn({0xf6f410?, 0x0})
	golang.org/x/net@v0.15.0/internal/socket/rawconn.go:57 +0x142
golang.org/x/net/ipv6.NewPacketConn({0xf6e758?, 0x0})
	golang.org/x/net@v0.15.0/ipv6/endpoint.go:121 +0x3d
github.com/hashicorp/mdns.(*client).setInterface(0xc000192900, 0xc000140768?)
	github.com/hashicorp/mdns@v1.0.5/client.go:222 +0xc5
github.com/hashicorp/mdns.Query(0xc000140808)
	github.com/hashicorp/mdns@v1.0.5/client.go:76 +0x8a
github.com/canonical/microcloud/microcloud/mdns.Lookup({0xf6b5a0, 0x155bd40}, 0xc000087da0, {0xc000191680, 0xd}, 0xc0001408a8?)
	github.com/canonical/microcloud/microcloud/mdns/lookup.go:151 +0x212
github.com/canonical/microcloud/microcloud/mdns.LookupPeers({0xf6b5a0, 0x155bd40}, 0xc000140bb8?, {0xe0fde8, 0x3}, {0xc00019160e, 0x2})
	github.com/canonical/microcloud/microcloud/mdns/lookup.go:64 +0x1b8
main.lookupPeers(0xc000087ec0, 0x1, 0x2?, 0xc0001879e0, {0x0, 0x0, 0xd?}, 0x7ffdace5b42c?)
	github.com/canonical/microcloud/microcloud/cmd/microcloud/main_init.go:217 +0x4bc
main.(*cmdInit).RunInteractive(0xc000145f80, 0x0?, {0x0?, 0x0?, 0x0?})
	github.com/canonical/microcloud/microcloud/cmd/microcloud/main_init.go:129 +0x49e
main.(*cmdInit).Run(0x0?, 0x0?, {0xc0001863f0?, 0x0?, 0x0?})
	github.com/canonical/microcloud/microcloud/cmd/microcloud/main_init.go:78 +0x36
github.com/spf13/cobra.(*Command).execute(0xc000004600, {0xc000085e40, 0x3, 0x4})
	github.com/spf13/cobra@v1.7.0/command.go:940 +0x87c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000004300)
	github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.7.0/command.go:992
main.main()
	github.com/canonical/microcloud/microcloud/cmd/microcloud/main.go:101 +0x8aa
gabrielmougard commented

@masnax I tried doing the following:

// in the `Lookup(ctx context.Context, iface *net.Interface, service string, size int) ([]*mdns.ServiceEntry, error)` function
...
params := mdns.DefaultParams(service)
params.Interface = iface
params.Entries = entriesCh
params.Timeout = 100 * time.Millisecond
if !ipv6Supported {
	params.DisableIPv6 = true
}

err := mdns.Query(params)
...

And the err := mdns.Query(params) is blocking forever... Don't really understand why (it works fine if I re-enable ipv6 on the machine).

I also tried to log a few things but I don't get much information:

// Still in the `Lookup` function, trying to inspect `entriesCh`
go func() {
		for {
			select {
			case <-ctx.Done():
				return
			default:
				fmt.Printf("gabrielm - mdns.Lookup - waiting for entries: %v\n", entriesCh)
				for entry := range entriesCh {
					entries = append(entries, entry)
				}
			}
		}
	}()

and I get:

gabrielm - mdns.Lookup - waiting for entries: 0xc00008d8602023/12/05 16:48:35 [INFO] mdns: Closing client {true false 0xc000064918 <nil> 0xc000064920 <nil> 1 0xc000042fc0}
gabrielm - mdns.Lookup - waiting for entries: 0xc00008d8602023/12/05 16:48:35 [INFO] mdns: Closing client {true false 0xc000064918 <nil> 0xc000064920 <nil> 1 0xc000042fc0}
...

More generally, how do you debug that kind of issue (I would probably need to check what's happening inside func (c *client) query(params *QueryParam) error of the mDNS lib we use) ? Thanks

roosterfish commented

@gabrielmougard you can try with a small client outside of the MicroCloud codebase to see if you can reproduce the blocking.

Invoke it with go run main.go --iface {youriface}:

package main

import (
	"flag"
	"net"
	"time"

	"github.com/hashicorp/mdns"
)

func main() {
	ifaceName := flag.String("iface", "eth0", "Interface to use")
	flag.Parse()

	iface, err := net.InterfaceByName(*ifaceName)
	if err != nil {
		panic(err)
	}

	entries := make(chan *mdns.ServiceEntry)
	params := &mdns.QueryParam{
		Service:     "_foobar._tcp",
		Timeout:     1 * time.Second,
		Interface:   iface,
		Entries:     entries,
		DisableIPv6: true,
	}
	params.Entries = entries

	err = mdns.Query(params)
	if err != nil {
		panic(err)
	}

	defer close(entries)
}
  • 👍1
gabrielmougard commented

@roosterfish I'll try that. thanks