Microcloud init - Error: Failed to get system resources of peer : No auth secret in response

[evoelise]

Originally raised in the LXD project so re-raising this here.
Also this was marked as a duplicate of #107 but reading that issue - this seems like a different issue/cause.

When trying to init microcloud in a 3 server cluster (physical servers not VMs) I get the following error:

Error: Failed to get system resources of peer "r520-03": No auth secret in response

I am using the edge version of lxd as I want to try this with the new lxd GUI.
I am using the latest version of microcloud on a fresh installation of Ubuntu 22.04.
Ubuntu and all snaps fully updated.

What can I do to diagnose this issue and resolve it?

To reproduce on 3 servers with a fresh install of Ubuntu Server 22.04

sudo snap install lxd
sudo snap refresh lxd --edge
sudo snap install microceph microovn microcloud

Then run the init command:

testuser@r520-02:~$ sudo lxd --version
4.0.9
testuser@r520-02:~$ sudo microcloud --version
0.1
testuser@r520-02:~$ sudo uname -a
Linux r520-02 5.15.0-72-generic #79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
testuser@r520-02:~$ sudo microcloud init
Using address "192.168.1.127" for MicroCloud
Limit search for other MicroCloud servers to 192.168.1.127/24? (yes/no) [default=yes]:
Scanning for eligible servers ...

 Selected "r520-03" at "192.168.1.128"
 Selected "r520-04" at "192.168.1.123"

Error: Failed to get system resources of peer "r520-04": No auth secret in response

This is shown in the syslog
May 20 11:12:27 r520-02 kernel: [ 4768.307666] audit: type=1400 audit(1684581147.188:146): apparmor="DENIED" operation="open" profile="snap.microcloud.microcloud" name="/var/lib/snapd/hostfs/etc/ssl/certs/ca-certificates.crt" pid=5993 comm="microcloud" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

More information that may be useful:

testuser@r520-02:~$ snap services
Service            Startup   Current   Notes
lxd.activate       enabled   inactive  -
lxd.daemon         enabled   active    socket-activated
microceph.daemon   enabled   active    -
microceph.mds      disabled  inactive  -
microceph.mgr      disabled  inactive  -
microceph.mon      disabled  inactive  -
microceph.osd      disabled  inactive  -
microceph.rgw      disabled  inactive  -
microcloud.daemon  enabled   active    -
microovn.central   disabled  inactive  -
microovn.chassis   disabled  inactive  -
microovn.daemon    enabled   active    -
microovn.switch    disabled  inactive  -

testuser@r520-02:~$ snap list
Name        Version        Rev    Tracking       Publisher   Notes
core20      20230503       1891   latest/stable  canonical✓  base
core22      20230503       634    latest/stable  canonical✓  base
lxd         git-407205d    23988  4.0/edge       canonical✓  -
microceph   0+git.fdf6d5e  338    latest/stable  canonical✓  -
microcloud  0+git.9cb7ccd  412    latest/stable  canonical✓  -
microovn    0+git.f8a4497  91     latest/stable  canonical✓  -
snapd       2.59.2         19122  latest/stable  canonical✓  snapd

[spanjeta]

Same problem