Update of raspberrypi-kernel fails `dpkg-divert: error: unable to change ownership of target file '/boot/...`
vogler opened this issue · 12 comments
Updates of raspberrypi-kernel and raspberrypi-bootloader on my RPi4 fail every time and leave me with a broken /boot.
I have no problems on my RPi3 (both running Raspbian 11). The only difference is that my RPi4 uses a 64bit kernel and boots from an SSD instead of an SD card.
Here's the log of an update:
...
The following packages will be upgraded:
broot containerd.io gh libksba8 linux-libc-dev raspberrypi-bootloader raspberrypi-kernel
rpi-eeprom tailscale vcdbg xserver-common xserver-xorg-core xvfb
...
Preparing to unpack .../05-raspberrypi-bootloader_1%3a1.20230106-1_armhf.deb ...
Adding 'diversion of /boot/start.elf to /usr/share/rpikernelhack/start.elf by rpikernelhack'
...
Setting up raspberrypi-kernel (1:1.20230106-1) ...
Removing 'diversion of /boot/kernel.img to /usr/share/rpikernelhack/kernel.img by rpikernelhack'
dpkg-divert: error: unable to change ownership of target file '/boot/kernel.img.dpkg-divert.tmp': Operation not permitted
dpkg: error processing package raspberrypi-kernel (--configure):
installed raspberrypi-kernel package post-installation script subprocess returned error exit status 2
Setting up broot (1.19.0) ...
Setting up containerd.io (1.6.15-1) ...
Setting up gh (2.21.2) ...
Setting up raspberrypi-bootloader (1:1.20230106-1) ...
Removing 'diversion of /boot/start.elf to /usr/share/rpikernelhack/start.elf by rpikernelhack'
dpkg-divert: error: unable to change ownership of target file '/boot/start.elf.dpkg-divert.tmp': Operation not permitted
dpkg: error processing package raspberrypi-bootloader (--configure):
installed raspberrypi-bootloader package post-installation script subprocess returned error exit status 2
...
dpkg: dependency problems prevent configuration of rpi-eeprom:
rpi-eeprom depends on raspberrypi-bootloader (>= 1.20190819); however:
Package raspberrypi-bootloader is not configured yet.
dpkg: error processing package rpi-eeprom (--configure):
dependency problems - leaving unconfigured
...
Errors were encountered while processing:
raspberrypi-kernel
raspberrypi-bootloader
rpi-eeprom
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
W: Operation was interrupted before it could finishAfterwards, /boot is broken:
$ ls -l /boot
total 28K
drwxr-xr-x 2 pi pi 24K Jan 12 09:19 overlays
-rw-rw-rw- 1 pi pi 80 Nov 16 2021 cmdline.txt
-rw-rw-rw- 1 pi pi 2.5K Nov 15 2021 config.txt
-rw-rw-rw- 1 pi pi 145 Sep 26 2019 issue.txt
$ ls -l /usr/share/rpikernelhack
total 49M
drwxr-xr-x 2 root root 36K Jan 12 09:19 overlays
-rw-r--r-- 1 root root 28K Jan 6 12:16 bcm2708-rpi-b.dtb
-rw-r--r-- 1 root root 28K Jan 6 12:16 bcm2708-rpi-b-plus.dtb
-rw-r--r-- 1 root root 28K Jan 6 12:16 bcm2708-rpi-b-rev1.dtb
-rw-r--r-- 1 root root 28K Jan 6 12:16 bcm2708-rpi-cm.dtb
-rw-r--r-- 1 root root 28K Jan 6 12:16 bcm2708-rpi-zero.dtb
-rw-r--r-- 1 root root 29K Jan 6 12:16 bcm2708-rpi-zero-w.dtb
-rw-r--r-- 1 root root 29K Jan 6 12:16 bcm2709-rpi-2-b.dtb
-rw-r--r-- 1 root root 29K Jan 6 12:16 bcm2709-rpi-cm2.dtb
-rw-r--r-- 1 root root 30K Jan 6 12:16 bcm2710-rpi-2-b.dtb
-rw-r--r-- 1 root root 32K Jan 6 12:16 bcm2710-rpi-3-b.dtb
-rw-r--r-- 1 root root 32K Jan 6 12:16 bcm2710-rpi-3-b-plus.dtb
-rw-r--r-- 1 root root 30K Jan 6 12:16 bcm2710-rpi-cm3.dtb
-rw-r--r-- 1 root root 31K Jan 6 12:16 bcm2710-rpi-zero-2.dtb
-rw-r--r-- 1 root root 31K Jan 6 12:16 bcm2710-rpi-zero-2-w.dtb
-rw-r--r-- 1 root root 52K Jan 6 12:16 bcm2711-rpi-400.dtb
-rw-r--r-- 1 root root 52K Jan 6 12:16 bcm2711-rpi-4-b.dtb
-rw-r--r-- 1 root root 52K Jan 6 12:16 bcm2711-rpi-cm4.dtb
-rw-r--r-- 1 root root 50K Jan 6 12:16 bcm2711-rpi-cm4s.dtb
-rw-r--r-- 1 root root 52K Jan 6 12:16 bootcode.bin
-rw-r--r-- 1 root root 19K Jan 6 12:16 COPYING.linux
-rw-r--r-- 1 root root 3.1K Jan 6 12:16 fixup4cd.dat
-rw-r--r-- 1 root root 5.3K Jan 6 12:16 fixup4.dat
-rw-r--r-- 1 root root 8.2K Jan 6 12:16 fixup4db.dat
-rw-r--r-- 1 root root 8.2K Jan 6 12:16 fixup4x.dat
-rw-r--r-- 1 root root 3.1K Jan 6 12:16 fixup_cd.dat
-rw-r--r-- 1 root root 7.1K Jan 6 12:16 fixup.dat
-rw-r--r-- 1 root root 10K Jan 6 12:16 fixup_db.dat
-rw-r--r-- 1 root root 10K Jan 6 12:16 fixup_x.dat
-rw-r--r-- 1 root root 6.4M Jan 6 12:16 kernel7.img
-rw-r--r-- 1 root root 6.8M Jan 6 12:16 kernel7l.img
-rw-r--r-- 1 root root 7.9M Jan 6 12:16 kernel8.img
-rw-r--r-- 1 root root 6.0M Jan 6 12:16 kernel.img
-rw-r--r-- 1 root root 1.6K Jan 6 12:16 LICENCE.broadcom
-rw-r--r-- 1 root root 787K Jan 6 12:16 start4cd.elf
-rw-r--r-- 1 root root 3.6M Jan 6 12:16 start4db.elf
-rw-r--r-- 1 root root 2.2M Jan 6 12:16 start4.elf
-rw-r--r-- 1 root root 2.9M Jan 6 12:16 start4x.elf
-rw-r--r-- 1 root root 787K Jan 6 12:16 start_cd.elf
-rw-r--r-- 1 root root 4.6M Jan 6 12:16 start_db.elf
-rw-r--r-- 1 root root 2.9M Jan 6 12:16 start.elf
-rw-r--r-- 1 root root 3.6M Jan 6 12:16 start_x.elf- If I
sudo umount /bootbefore the update, there are no problems during the update and I can afterwardscopy -athe new files from sda2 to sda1 and mount sda1 as /boot again. - If I missed
sudo umount /bootbefore the update, Icp -a /usr/share/rpikernelhack/* /boot/but then still need tosudo umount /bootfor the update to go through.
Further info:
$ uname -a
Linux rpi4 5.15.76-v8+ #1597 SMP PREEMPT Fri Nov 4 12:16:41 GMT 2022 aarch64 GNU/Linux
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
...
sda 8:0 0 465.8G 0 disk
├─sda1 8:1 0 256M 0 part /boot
└─sda2 8:2 0 465.5G 0 part /media/usb0
$ cat /etc/fstab
proc /proc proc defaults 0 0
PARTUUID=2af3726c-01 /boot vfat defaults,flush 0 2
PARTUUID=2af3726c-02 / ext4 defaults,noatime 0 1
# a swapfile is not a swap partition, no line here
# use dphys-swapfile swap[on|off] for thathttps://raspberrypi.stackexchange.com/questions/51410/what-is-rpikernelhack
Unable to reproduce the issue here. How do I get to the same state starting with a clean official image? The fact that your /boot files are owned by 'pi' indicated that it was mounted in a strange way.
My assumption is that it has to do with how I moved from SD card to SSD (also for boot), but I don't know what should be wrong about the setup.
I think I followed this at the time: https://jamesachambers.com/raspberry-pi-4-usb-boot-config-guide-for-ssd-flash-drives/
$ ls -l /
total 84
drwxr-xr-x 2 root root 4096 Dec 20 16:57 bin
drwxr-xr-x 3 pi pi 4096 Jan 1 1970 boot
...Is there some other place that's relevant besides /etc/fstab?
What's the output of mount?
$ mount
/dev/sda2 on / type ext4 (rw,noatime,stripe=8191)
devtmpfs on /dev type devtmpfs (rw,relatime,size=1776440k,nr_inodes=444110,mode=755)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=776884k,nr_inodes=819200,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
/var/lib/snapd/snaps/certbot_2616.snap on /snap/certbot/2616 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core18_2657.snap on /snap/core18/2657 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/certbot_2580.snap on /snap/certbot/2580 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core_14402.snap on /snap/core/14402 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core20_1739.snap on /snap/core20/1739 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core22_446.snap on /snap/core22/446 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core18_2670.snap on /snap/core18/2670 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/nvim_2785.snap on /snap/nvim/2785 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/dev/sda2 on /media/usb0 type ext4 (rw,nodev,noexec,noatime,nodiratime,stripe=8191)
/dev/sda1 on /media/usb1 type vfat (rw,nodev,noexec,noatime,nodiratime,sync,uid=1000,gid=1000,fmask=0111,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=388440k,nr_inodes=97110,mode=700,uid=1000,gid=1000)
nsfs on /run/docker/netns/default type nsfs (rw)
/var/lib/snapd/snaps/core22_472.snap on /snap/core22/472 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core20_1781.snap on /snap/core20/1781 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/nvim_2794.snap on /snap/nvim/2794 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core_14449.snap on /snap/core/14449 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/dev/sda1 on /boot type vfat (rw,relatime,sync,uid=1000,gid=1000,fmask=0111,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)I believe the uid=1000,gid=1000 part is the problem, but I don't know where it's coming from.
/dev/sda2 on / type ext4 (rw,noatime,stripe=8191)
/dev/sda2 on /media/usb0 type ext4 (rw,nodev,noexec,noatime,nodiratime,stripe=8191)
/dev/sda1 on /media/usb1 type vfat (rw,nodev,noexec,noatime,nodiratime,sync,uid=1000,gid=1000,fmask=0111,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
/dev/sda1 on /boot type vfat (rw,relatime,sync,uid=1000,gid=1000,fmask=0111,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
Is it "expected" for these to appear multiple times?
I don't need the additional /media mounts and don't know where they're coming from, but I guess it shouldn't be a problem since there's no problem with sda2 being mounted twice.
I found that the /media mounts are created by usbmount:
$ cat /etc/usbmount/usbmount.conf
# Configuration file for the usbmount package, which mounts removable
# storage devices when they are plugged in and unmounts them when they
# are removed.
# Change to zero to disable usbmount
ENABLED=1
# Mountpoints: These directories are eligible as mointpoints for
# removable storage devices. A newly plugged in device is mounted on
# the first directory in this list that exists and on which nothing is
# mounted yet.
MOUNTPOINTS="/media/usb0 /media/usb1 /media/usb2 /media/usb3
/media/usb4 /media/usb5 /media/usb6 /media/usb7"
...
FS_MOUNTOPTIONS="-fstype=vfat,gid=pi,uid=pi,dmask=0022,fmask=0111"I compared with my RPi3 where /boot is owned by root instead of pi (both when mounted and umounted)
whereas on my RPi4 it is owned by pi when mounted (but by root when unmounted).
Seems like usbmount's FS_MOUNTOPTIONS influenced the /boot mount defined in /etc/fstab.
After disabling usbmount and rebooting, /boot is now owned by root, also when mounted.
I'll wait for another update involving raspberrypi-kernel to test and open this issue again if the above did not fix it.
Thanks!
Great detective work
@XECDesign Based on the description above, would this problem affect everyone booting from a USB device, e.g. a USB3 SSD? (although if it does, it seems like this problem should have cropped up before??)
I wonder if it'd be possible to somehow use https://github.com/rbrito/usbmount#hook-scripts to prevent usbmount "double mounting" the boot and root partitions? ping also @spl237
Aha, seems to be a known problem? rbrito/usbmount#35
affect everyone booting from a USB device, e.g. a USB3 SSD? (although if it does, it seems like this problem should have cropped up before??)
If they also installed usbmount, no? I don't have it on my RPi3, but also don't remember installing it on my RPi4.
I know nothing about the internals, but I found it very weird that usbmount's options also apply to the mount defined in /etc/fstab. Can't you mount the same device with different options? Why don't the options in /etc/fstab have a higher priority or why don't subsequent mounts with conflicting options fail instead of overwriting others?
would this problem affect everyone booting from a USB device
Not out of the box, no. This will be resolved when we move to different packaging for the firmware that doesn't rely on the dpkg-divert mechanism.