Failed to automatically find an unused IPv4 subnet, manual configuration required
Closed this issue · 8 comments
IsaacJT commented
I'm trying to set up a cluster on three Orange Pi 5+'s running Ubuntu Core 22 and running into an issue when running microcloud init.
itrue@orangepi-2:~$ sudo microcloud init
Waiting for LXD to start...
Using address "10.42.0.49" for MicroCloud
Limit search for other MicroCloud servers to 10.42.0.49/8? (yes/no) [default=yes]:
Scanning for eligible servers ...
Selected "orangepi-1" at "10.42.0.198"
Selected "orangepi-3" at "10.42.0.217"
Selected "orangepi-2" at "10.42.0.49"
Would you like to set up local storage? (yes/no) [default=yes]: no
Would you like to set up distributed storage? (yes/no) [default=yes]:
Select from the available unpartitioned disks:
Select which disks to wipe:
Using 1 disk(s) on "orangepi-2" for remote storage pool
Using 1 disk(s) on "orangepi-1" for remote storage pool
Using 1 disk(s) on "orangepi-3" for remote storage pool
Configure distributed networking? (yes/no) [default=yes]:
Select exactly one network interface from each cluster member:
Using "enP4p65s0" on "orangepi-3" for OVN uplink
Using "enP4p65s0" on "orangepi-2" for OVN uplink
Using "enP4p65s0" on "orangepi-1" for OVN uplink
Specify the IPv4 gateway (CIDR) on the uplink network (empty to skip IPv4): 192.168.123.1/24
Specify the first IPv4 address in the range to use with LXD: 192.168.123.50
Specify the last IPv4 address in the range to use with LXD: 192.168.123.250
Specify the IPv6 gateway (CIDR) on the uplink network (empty to skip IPv6):
Initializing a new cluster
Local MicroCloud is ready
Local LXD is ready
Local MicroOVN is ready
Local MicroCeph is ready
Awaiting cluster formation ...
Peer "orangepi-1" has joined the cluster
Peer "orangepi-3" has joined the cluster
Configuring cluster-wide devices ...
Error: Failed generating auto config: Failed to automatically find an unused IPv4 subnet, manual configuration required
IP config of orangepi-1:
itrue@orangepi-1:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enP3p49s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether c0:74:2b:ff:42:79 brd ff:ff:ff:ff:ff:ff
inet 10.42.0.198/8 metric 100 brd 10.255.255.255 scope global dynamic enP3p49s0
valid_lft 3257sec preferred_lft 3257sec
inet6 fe80::c274:2bff:feff:4279/64 scope link
valid_lft forever preferred_lft forever
3: enP4p65s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether c0:74:2b:ff:42:78 brd ff:ff:ff:ff:ff:ff
inet6 fe80::c274:2bff:feff:4278/64 scope link
valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 06:32:5f:21:b9:23 brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether e2:9c:1c:8e:01:f6 brd ff:ff:ff:ff:ff:ff
6: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
link/ether b6:0b:69:70:56:3c brd ff:ff:ff:ff:ff:ff
inet6 fe80::449b:58ff:fe55:45c7/64 scope link
valid_lft forever preferred_lft forever
itrue@orangepi-1:~$ ip r
default via 10.42.0.1 dev enP3p49s0 proto dhcp src 10.42.0.198 metric 100
10.0.0.0/8 dev enP3p49s0 proto kernel scope link src 10.42.0.198 metric 100
10.42.0.1 dev enP3p49s0 proto dhcp scope link src 10.42.0.198 metric 100
IP config of orangepi-2:
itrue@orangepi-2:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enP3p49s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether c0:74:2b:ff:3a:db brd ff:ff:ff:ff:ff:ff
inet 10.42.0.49/8 metric 100 brd 10.255.255.255 scope global dynamic enP3p49s0
valid_lft 3215sec preferred_lft 3215sec
inet6 fe80::c274:2bff:feff:3adb/64 scope link
valid_lft forever preferred_lft forever
3: enP4p65s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether c0:74:2b:ff:3a:da brd ff:ff:ff:ff:ff:ff
inet6 fe80::c274:2bff:feff:3ada/64 scope link
valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether de:05:47:8d:c7:27 brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether be:e1:a1:a0:74:b1 brd ff:ff:ff:ff:ff:ff
6: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
link/ether ce:13:87:ac:76:8b brd ff:ff:ff:ff:ff:ff
inet6 fe80::9842:c4ff:feb9:bd76/64 scope link
valid_lft forever preferred_lft forever
itrue@orangepi-2:~$ ip r
default via 10.42.0.1 dev enP3p49s0 proto dhcp src 10.42.0.49 metric 100
10.0.0.0/8 dev enP3p49s0 proto kernel scope link src 10.42.0.49 metric 100
10.42.0.1 dev enP3p49s0 proto dhcp scope link src 10.42.0.49 metric 100
IP config of orangepi-3:
itrue@orangepi-3:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enP3p49s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether c0:74:2b:ff:39:51 brd ff:ff:ff:ff:ff:ff
inet 10.42.0.217/8 metric 100 brd 10.255.255.255 scope global dynamic enP3p49s0
valid_lft 3238sec preferred_lft 3238sec
inet6 fe80::c274:2bff:feff:3951/64 scope link
valid_lft forever preferred_lft forever
3: enP4p65s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether c0:74:2b:ff:39:50 brd ff:ff:ff:ff:ff:ff
inet6 fe80::c274:2bff:feff:3950/64 scope link
valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 22:8f:3a:9c:78:6b brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 2a:30:d1:71:c0:23 brd ff:ff:ff:ff:ff:ff
6: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
link/ether 72:e0:09:9a:0a:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::9ca0:f5ff:fed8:2fcb/64 scope link
valid_lft forever preferred_lft forever
itrue@orangepi-3:~$ ip r
default via 10.42.0.1 dev enP3p49s0 proto dhcp src 10.42.0.217 metric 100
10.0.0.0/8 dev enP3p49s0 proto kernel scope link src 10.42.0.217 metric 100
10.42.0.1 dev enP3p49s0 proto dhcp scope link src 10.42.0.217 metric 100
IsaacJT commented
I've caused this by inadvertently putting the uplink network range in the same subnet as the management interface. Perhaps there could be some validation done during the setup wizard to check that this doesn't happen?
tomponline commented
@masnax is this something we can do?
IsaacJT commented
Sorry for the back and forth - re-opening this as it's is still happening even with different subnets. I updated the original comment with the new info.
masnax commented
So LXD determines a random unused /24 subnet from within 10.0.0.0/8 when configuring the OVN network: https://github.com/canonical/lxd/blob/8d0390b1e475728717181499ea56ce04a59c5cba/lxd/network/network_utils.go#L569-L589. If it can't find one, then it returns that error.
In the effort of not having too many required questions in the MicroCloud setup, I think what we might be able to do here is export those functions in LXD (under shared) and then check if there's an unused subnet when setting up the OVN network in MicroCloud, and if there isn't then we can prompt the user to enter a subnet themselves. So basically just fallback to user input if auto config fails. Does that sound reasonable @tomponline?
masnax commented
Alternatively, we can update that function to cover more private IP blocks (192.168.0.0/16, 172.16.0.0/12)
tomponline commented