microceph does not allow adding partitioned disks

[pedrofragola]

Based on issue [0], I tested snapd 2.61 in the edge channel to be able to use disks with partitions (/dev/sdb1). However, I still encountered the issue:

sudo microceph disk add /dev/sdb1
Error: Failed adding new disk: Failed to bootstrap OSD: Failed to run: ceph-osd --mkfs --no-mon-config -i 1: exit status 250 (2023-10-20T19:50:04.885+0000 7fd2a09e7800 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (13) Permission denied
2023-10-20T19:50:04.885+0000 7fd2a09e7800 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (13) Permission denied
2023-10-20T19:50:04.885+0000 7fd2a09e7800 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (13) Permission denied
2023-10-20T19:50:04.889+0000 7fd2a09e7800 -1 bluestore(/var/lib/ceph/osd/ceph-1) _setup_block_symlink_or_file failed to open block file: (13) Permission denied
2023-10-20T19:50:04.889+0000 7fd2a09e7800 -1 bluestore(/var/lib/ceph/osd/ceph-1) mkfs failed, (13) Permission denied
2023-10-20T19:50:04.889+0000 7fd2a09e7800 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied
2023-10-20T19:50:04.889+0000 7fd2a09e7800 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1: (13) Permission denied)

After discussing this internally with the microcloud team, we added the following line to the file /var/lib/snapd/apparmor/profiles/snap.microceph.daemon:

/dev/sd{,[a-z]}[a-z][0-9]{,[0-9],[0-9][0-9]} rwk, # SCSI rule!

Then we ran the following commands:

sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.microceph.daemon
sudo snap restart microceph

After this, it was possible to use partitions. Ideally, there should be a fix for this and the apparmor should already have the correct file.

[0] canonical/snapd#13150

[sabaini]

Hi @pedrofragola ftr. the linked snapd patch turned out to be erroneous, we will need to come up with a new interface for snapd

[pedrofragola]

Hi @sabaini thanks for the info... do you will file a new snapd bug? if so let me know the link to follow

[itoffshore]

I worked around this issue here

• put apparmor in complain mode so microceph init works
• copy line 619 in /var/lib/snapd/apparmor/profiles/snap.microceph.osd & create an entry for your partition (e.g /dev/vda3 rwk, - so the OSD's start after a reboot) - & make the profile immutable (for the time being until this is fixed)

[root@host1 ~]# microceph.ceph status
  cluster:
    id:     61ee0596-5913-48c2-92dd-7d24d74bd979
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum host1,host3,host4 (age 39m)
    mgr: host1(active, since 62m), standbys: host4, host3
    osd: 4 osds: 4 up (since 18m), 4 in (since 41m)
 
  data:
    pools:   1 pools, 1 pgs
    objects: 2 objects, 449 KiB
    usage:   84 MiB used, 152 GiB / 152 GiB avail
    pgs:     1 active+clean

---

I also played around with cephadm yesterday to run a ceph cluster inside podman - but this forces you to install OSD's on top of lvm lv's rather than directly onto partitions. The mgr service also kept crashing (possibly due to using podman rather than docker)

microceph is much nicer & I'm going back to it today (it always gave me a healthy cluster)

[adam-vest]

Howdy!

I stumbled upon this bug report when I was deploying microceph locally as well, as my original design plan had intended on using partitions where necessary (some devices I'm using only have one storage device). I did confirm that modifying the apparmor profiles allowed things to connect smoothly; however, chattr-ing those files as immutable isn't really a sustainable solution, since if the host ever needs to reboot (or the snap gets restarted, updated, etc), microceph will just fail to come up because snap can't control those files in the way it wants.

To work around this problem for now, I instead wrote a bash script that runs as an ExecStartPre on the snap.microceph.daemon & snap.microceph.osd systemd services. Script is as such (making sure it's executable):

#!/bin/bash
if [[ ! $(grep Cephy /var/lib/snapd/apparmor/profiles/snap.microceph.daemon) ]]; then
        sed -i '/loopback control$/a \/dev\/dm-\[0-9\] rwk,\t\t\t\t\t\t# Cephy' /var/lib/snapd/apparmor/profiles/snap.microceph.daemon;
        apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.microceph.daemon;
fi

if [[ ! $(grep Cephy /var/lib/snapd/apparmor/profiles/snap.microceph.osd) ]]; then
        sed -i '/loopback control$/a \/dev\/dm-\[0-9\] rwk,\t\t\t\t\t\t# Cephy' /var/lib/snapd/apparmor/profiles/snap.microceph.osd;
        apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.microceph.osd;
fi

(In my case I'm using LVM volumes, so dm-[0-9] made sense for me - replace as appropriate for your needs)

Then do sudo systemctl edit on both of the above-named systemd services, and add the following:

[Service]
ExecStartPre=/usr/local/bin/microceph.sh

(or whatever path you placed the script in)

Then sudo systemctl daemon-reload for good measure, and at this point microceph should be able to work with partitions sustainably until the snap team makes this support native.

[itoffshore]

@adam-vest - many thanks for a better solution - you can also --drop-in (override) service scripts with:

• systemctl edit unit --drop-in=drop_in_name

systemctl edit snap.microceph.daemon --drop-in=override
systemctl edit snap.microceph.osd --drop-in=override

• see Drop-In files

[UtkarshBhatthere]

Thank you @adam-vest and @itoffshore for sharing workarounds.

[itoffshore]

@UtkarshBhatthere @adam-vest - here's the script I use:

• For a shell script called by a service it's probably a good idea to exit 0
• There are 3 x snap profiles with a configuration for virtio (see 2 x commented lines in the script below to use all 3)

For the override I only needed to:

• systemctl edit snap.microceph.osd --drop-in=override
• systemctl daemon-reload

[Service]
ExecStartPre=/path/to/script

• In the script below just change $ADD to whatever rule makes sense for your partitions
• As sed is inserting a line you shouldn't need to escape delimiters in $ADD

#!/bin/sh

TAG="Cephy"
ADD="/dev/vda[4-9] rwk,\t\t\t\t\t   # $TAG"
SEARCH="/dev/vd\[a-z\]"
#PROFILES="/var/lib/snapd/apparmor/profiles/snap.microceph*"
#FILES=$(grep -l $SEARCH $PROFILES)
FILES="/var/lib/snapd/apparmor/profiles/snap.microceph.osd"

for file in $FILES; do
        if ! grep -q $TAG $file; then
                line=$(grep -n $SEARCH $file | cut -d : -f 1)
                sed -i "$line i $ADD" $file
                echo "Reloading: $file"
                apparmor_parser -r $file
        else
                echo "Already configured: $file"
        fi
done

exit 0

• Fixing all 3 x snap profiles didn't show any benefits - microceph init still didn't display any choices for partitions.