3rd party kernel installation always fails on f39 silverblue

[TrixieUA]

Host system details

Tested in boxes vm with fedora 39 Silverblue image.

Provide the output of rpm-ostree status.

State: idle
Deployments:
● fedora:fedora/39/x86_64/silverblue
                  Version: 39.20231013.n.0 (2023-10-13T08:11:38Z)
               BaseCommit: 0bc450e922e7f9fb0fab9c54d673cfb94b80105f3e7bcab0920144d3912318df
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: langpacks-en

  fedora:fedora/39/x86_64/silverblue
                  Version: 39.20231002.n.0 (2023-10-02T08:04:13Z)
               BaseCommit: 1bf3b61a255190f422183cafa02b32bd05044fa0ae784ae8608d6ce4213ca71e
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: langpacks-en

Expected vs actual behavior

Expected (output of f38 Silverblue)

# rpm-ostree override remove kernel kernel-core kernel-modules kernel-modules-core kernel-modules-extra --install kernel-cachyos-bore-eevdf
Checking out tree 3edbf75... done
Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora copr:copr.fedorainfracloud.org:bieszczaders:kernel-cachyos updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2023-03-14T10:56:46Z solvables: 4
rpm-md repo 'fedora-modular' (cached); generated: 2023-04-13T20:30:47Z solvables: 1082
rpm-md repo 'updates-modular' (cached); generated: 2023-09-16T00:25:03Z solvables: 1087
rpm-md repo 'updates' (cached); generated: 2023-10-13T01:47:25Z solvables: 25185
rpm-md repo 'fedora' (cached); generated: 2023-04-13T20:37:10Z solvables: 69222
rpm-md repo 'copr:copr.fedorainfracloud.org:bieszczaders:kernel-cachyos' (cached); generated: 2023-10-13T14:38:28Z solvables: 77
rpm-md repo 'updates-archive' (cached); generated: 2023-10-10T02:42:32Z solvables: 41995
Resolving dependencies... done
Importing packages... done
Applying 5 overrides and 3 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Generating initramfs... done
Writing OSTree commit... done
Staging deployment... done
Removed:
  kernel-6.5.6-200.fc38.x86_64
  kernel-core-6.5.6-200.fc38.x86_64
  kernel-modules-6.5.6-200.fc38.x86_64
  kernel-modules-core-6.5.6-200.fc38.x86_64
  kernel-modules-extra-6.5.6-200.fc38.x86_64
Added:
  kernel-cachyos-bore-eevdf-6.5.7-cbe2.0.fc38.x86_64
  kernel-cachyos-bore-eevdf-core-6.5.7-cbe2.0.fc38.x86_64
  kernel-cachyos-bore-eevdf-modules-6.5.7-cbe2.0.fc38.x86_64
Use "rpm-ostree override reset" to undo overrides
Run "systemctl reboot" to start a reboot

After reboot system boot into new cachyos kernel

Actual(output of f39 Silverblue):

# rpm-ostree override remove kernel kernel-core kernel-modules kernel-modules-core kernel-modules-extra --install kernel-cachyos-bore-eevdf 
Checking out tree 0bc450e... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora copr:copr.fedorainfracloud.org:bieszczaders:kernel-cachyos updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2023-03-14T10:57:01Z solvables: 4
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2023-10-12T09:54:16Z solvables: 70830
rpm-md repo 'copr:copr.fedorainfracloud.org:bieszczaders:kernel-cachyos' (cached); generated: 2023-10-13T15:23:44Z solvables: 77
rpm-md repo 'updates-archive' (cached); generated: 2023-04-22T20:18:51Z solvables: 0
Resolving dependencies... done
Applying 5 overrides and 10 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
error: Running %post for kernel-cachyos-bore-eevdf-core: bwrap(/bin/sh): Child process killed by signal 1; run `journalctl -t 'rpm-ostree(kernel-cachyos-bore-eevdf-core.post)'` for more information
# journalctl -t 'rpm-ostree(kernel-cachyos-bore-eevdf-core.post)'
Sep 19 18:05:15 fedora rpm-ostree(kernel-cachyos-bore-eevdf-core.post)[3865]: Failed to chase '/boot': No such file or directory
-- Boot c275685080a84cde9a9fd7fb45619cde --
Sep 19 20:16:12 fedora rpm-ostree(kernel-cachyos-bore-eevdf-core.post)[3515]: Failed to chase '/boot': No such file or directory
Sep 19 20:16:43 fedora rpm-ostree(kernel-cachyos-bore-eevdf-core.post)[3596]: Failed to chase '/boot': No such file or directory
Sep 19 20:19:15 fedora rpm-ostree(kernel-cachyos-bore-eevdf-core.post)[3860]: Failed to chase '/boot': No such file or directory
Sep 19 20:20:29 fedora rpm-ostree(kernel-cachyos-bore-eevdf-core.post)[3954]: Failed to chase '/boot': No such file or directory
-- Boot 4ffc8539b71e4a2f949d2dc96cf550aa --
Oct 04 01:06:41 fedora rpm-ostree(kernel-cachyos-bore-eevdf-core.post)[3652]: Failed to chase '/boot': No such file or directory
-- Boot 7639acdeb50e4fcab53fdcdf38dd752f --
Oct 13 22:29:57 fedora rpm-ostree(kernel-cachyos-bore-eevdf-core.post)[4085]: Failed to chase '/boot': No such file or directory
-- Boot 5cc2575feee243d6a2379c59405424ed --
Oct 13 23:58:46 fedora rpm-ostree(kernel-cachyos-bore-eevdf-core.post)[3619]: Failed to chase '/boot': No such file or directory
...

rpm-ostree fails in %post for kernel-cachyos-bore-eevdf-core which is as far as i know is /bin/kerne-install %kver /usr/lib/modules/%kver/vmlinuz

Steps to reproduce it

Follow the instructions in this copr

Would you like to work on the issue?

I can't do nothing with rpm-ostree code but i am maintainer of specfile for this kernel so i will gladly try to do anything i can with specfile which can be found here

[pesader]

Hey! From my understanding, this happens because the posttrans scripts assume they can use certain programs that are replaced by rpm-ostree on Fedora Silverblue (e.g. dracut is replaced by rpm-ostree initramfs).

I reported this on the issue tracker, and you can check it out here here. Also, an attempt to fix this is on the way! If you can help with testing, it might land sooner 😉

[TrixieUA]

Hey! From my understanding, this happens because the posttrans scripts assume they can use certain programs that are replaced by rpm-ostree on Fedora Silverblue (e.g. dracut is replaced by rpm-ostree initramfs).

kernel-cachyos specfile doesnt use dracut directly, the command that fails is /bin/kernel-install add %{kverstr} /lib/modules/%{kverstr}/vmlinuz if there is rpm-ostree alternative to that i'll be glad to test.

Also thanks for the info for depmod and dracut because that will be helpfull later.