Systemd extension breaks steamos-readonly functionality

[gustakasn0v]

Context

Steam Deck comes with a readonly filesystem by default, to simplify updates and security. In https://help.steampowered.com/en/faqs/view/671A-4453-E8D2-323C Valve shows you can disable this functionality by running steamos-readonly disable. I do this to allow rclone mounts using /etc/fstab, as described here (ie. sudo ln -s ~/.bin/rclone /sbin/mount.rclone)

Bug

After running tailscale.sh, the /usr partition remains readonly when running steamos-readonly disable, even though the / partition is writeable:

(deck@steamdeck ~)$ sudo steamos-readonly disable
[sudo] password for deck: 
(deck@steamdeck ~)$ sudo steamos-readonly status
disabled
(1)(deck@steamdeck ~)$ sudo touch /test.txt
(deck@steamdeck ~)$ sudo touch /usr/test.txt
touch: cannot touch '/usr/test.txt': Read-only file system

Root cause and workaround

The issue seems to be because systemd-sysext mounts a readonly overlay on the /usr directory. I'm not sure if this is the default behaviour, or something configured in the tailscaled extension

(130)(deck@steamdeck ~)$ mount | grep usr
sysext on /usr type overlay (ro,relatime,lowerdir=/run/systemd/sysext/meta/usr:/run/systemd/sysext/extensions/tailscale/usr:/usr,index=off)

This seems to interfere with how steamos-readonly disable detects partitions that need rerouting, although I'm not 100% sure as I haven't looked at its implementation (the file is located in /usr/bin/steamos-readonly if someone wants to inspect it)

Stopping systemd-sysext unmounts the overlay and fixes the issue.

(1)(deck@steamdeck ~)$ systemctl stop systemd-sysext.service --now 
(deck@steamdeck ~)$ sudo touch /usr/test.txt
(deck@steamdeck ~)$

After doing the changes I needed to do, I re-enabled readonly mode via steamos-readonly enable, started systemd-sysext via systemctl start systemd-sysext.service --now, restarted tailscaled via systemctl restart tailscaled.service --now, and everything seems to be working fine for now

Next steps

1. Is it possible to make the tailscaled extension not disrupt steamos-readonly? Or is this systemd-sysext behaviour that's causing this?
2. Instead of using a systemd extension to achieve persistence across software updates, isn't it a better idea to install files in the user's home directory, and use systemd user services? Here is the code where EmuDeck, the popular emulation software on Steam Deck, achieves this

[legowerewolf]

This extension does nothing specific regarding steamos-readonly. It's kinda piggybacking on the out-of-box-configured overlay of /var and /etc to persist itself. I think it's systemd-sysext that's causing this.

Honestly, I didn't try implementing this as a user service. This thing got started based off that blog post (linked in readme) and then slowly refactored from there. I do wonder, though, would a user service have the permissions necessary to manage networking? This'll require study and experimentation.

[gustakasn0v]

1. Apologies, perhaps I didn't explain myself correctly. I'm not suggesting this script is messing with the steamos-readonly tool. But one side-effect of this script is indeed creating a read-only mount in the /usr directory, which doesn't exist if this script isn't run. Do you know if the Tailscale system extension (which I couldn't find in this repo) specifies any mount in /usr, or if this is systemd-sysext behaviour?
2. Agreed in moving to user-level services, this would probably require a deep-dive to see if it's viable

[legowerewolf]

The extension does partially overlay the /usr directory - this how we create its folder structure:

# create our target directory structure
mkdir -p tailscale/usr/{bin,sbin,lib/{systemd/system,extension-release.d}}

Other than this, there's nothing /usr specific.

[gustakasn0v]

I see, thanks for confirming. One of the things I've noticed is that using systemd-sysext seems to modify the /sbin mountpoint, even though it's only supposed to allow modifying the /usr and /opt mountpoints.

(deck@steamdeck ~)$ sudo steamos-readonly status
[sudo] password for deck: 
disabled
(130)(deck@steamdeck ~)$ sudo touch /sbin/test.txt
touch: cannot touch '/sbin/test.txt': Read-only file system
(1)(deck@steamdeck ~)$ df -h /sbin
Filesystem      Size  Used Avail Use% Mounted on
sysext          5.8G   12K  5.8G   1% /usr
(deck@steamdeck ~)$ systemd-sysext status
HIERARCHY EXTENSIONS SINCE                      
/opt      none       -                          
/usr      tailscale  Tue 2023-10-03 14:25:52 IST
(deck@steamdeck ~)$ systemctl stop systemd-sysext.service --now 
(deck@steamdeck ~)$ df -h /sbin
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p5  5.0G  3.3G  1.5G  69% /
(deck@steamdeck ~)$ sudo touch /sbin/test.txt
(deck@steamdeck ~)$ 

I'll ask/research around to see if this is expected. https://blogs.igalia.com/berto/2022/09/13/adding-software-to-the-steam-deck-with-systemd-sysext/ seems like a good place to start. But this is looking to be a systemd-sysext problem; perhaps there should at least be a warning in the README?

[gustakasn0v]

Eureka!

ls -l /   
...
lrwxrwxrwx   1 root root     7 Dec  2 09:30 sbin -> usr/bin 

The systemd ebuild currently expects sbin to point to usr/sbin not usr/bin and dies because of this mismatch.
https://forums.gentoo.org/viewtopic-p-8762999.html

This got me thinking, and indeed on Steam Deck /sbin/ points to /usr/bin

(deck@steamdeck ~)$ sudo ls -lh /sbin
lrwxrwxrwx 1 root root 7 Nov  8  2022 /sbin -> usr/bin

So that explains it. Up to you how you'd like to proceed; I'd suggest adding a note in the README to explain which partitions get unexpectedly made read-only besides /usr and /var (ls -lh / reveals /bin /lib /lib64 /mnt and /sbin will be made read-only as they link to /usr and /var)

[legowerewolf]

Hm. Well it'd be nice if Valve would fix this, but yeah, I'll see about adding a warning. Hopefully switching to a user service will fix this, but I need to get a testing environment set up for that.

[bertogg]

Hello, as I said in the blog post that you linked:

"When extensions are enabled the /usr and /opt directories become read-only because they are now part of an overlayfs. They will remain read-only even if you run steamos-readonly disable !!. If you really want to make the rootfs read-write you need to disable the extensions (systemd-sysext unmerge) first."

So it should be enough with something like this (I'm writing it from memory):

$ systemd-sysext unmerge
$ steamos-readonly disable
[ make your changes to the rootfs here ]
$ steamos-readonly enable
$ systemd-sysext merge

Is your question "It would be nice if steamos-readonly would do that automatically" ? Technically it's possible but when you do unmerge you make things "disappear" from the root filesystem so now you have side effects.

You can create your own script to do that and store it in your home directory or something.

[gustakasn0v]

Thanks @bertogg, the missing piece here was realising that SteamOS implements the /usr merge described in https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ (thanks for sharing this link in your article btw). So I was surprised when after running steamos-readonly disable I was still unable to modify /sbin. But now it makes sense

[legowerewolf]

Alright, so remaining to close this is just adding to the readme a warning and workaround steps? The floor's open if someone wants to make a PR for it.

I'll also file an issue to research (and, if possible, switch to) using a user service.