Rebase to Fedora Silverblue 40, cleanup removing layered deployment and re-doing rebase causes invalid rpm cache?

[rugk]

Describe the bug

When trying to rebase to Fedora Silverblue 40, deleting the deployment via rpm-ostree cleanup -p and re-doing it later (after an intermediate deployment on Fedora 39), fails with weird GPG/rpm signature errors…

The workaround is to delete the cache via rpm-ostree cleanup -bm, afterwards the rebase works successfully.

Reproduction steps

First, update to Fedora Silverblue 39.20240325.0 from Fedora 39 (39.20240417.0). In my case I did not boot into it, because of a secure boot issue (now resolved).
Go back to Fedora 39.20240417.0 (aka just boot into it).
rpm-ostree reset there (BTW I had been searching for rpm-ostree reset --revert 🙃) and rpm-ostree update trying to fix the issue…
At some point, decide to rebase to Fedora 40 again trying to find out whether the boot issue persists.
The issue persists, but you find out you can boot into it with Secure Boot disabled, so boot into it. (I dunno if it is relevant.)
Anyway, boot into the old working 39.20240417.0 again.

Now, you need to revert the bad try with rpm-ostree reset, which you do with rpm-ostree cleanup -p.
Then you rpm-ostree upgrade to 39.20240429.0 and also pin it. (Also removing mozilla-openh264 and booting into 39.20240429.0 again in a new deployment)

Now rebase again to Fedora 40 (that you just had deleted), …

Expected behavior

Rebase should work, it has worked before…

Actual behavior

Now rebase again to Fedora 40 (that you just had deleted), which fails like this:

$ rpm-ostree rebase fedora:fedora/40/x86_64/silverblue
2 metadata, 0 content objects fetched; 788 B transferred in 4 seconds; 0 Bytes content written
Checking out tree 7d06768... done
Enabled rpm-md repositories: fedora rpmfusion-free fedora-cisco-openh264 updates rpmfusion-free-updates updates-archive
Importing rpm-md... done
rpm-md repo 'fedora' (cached); generated: 2024-04-19T08:53:31Z solvables: 74873
rpm-md repo 'rpmfusion-free' (cached); generated: 2024-04-20T12:11:51Z solvables: 422
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2023-12-11T14:43:50Z solvables: 4
rpm-md repo 'updates' (cached); generated: 2024-04-29T01:08:34Z solvables: 8720
rpm-md repo 'rpmfusion-free-updates' (cached); generated: 2024-04-26T21:25:38Z solvables: 18
rpm-md repo 'updates-archive' (cached); generated: 2024-04-29T01:21:28Z solvables: 6905
Resolving dependencies... done
Will download: 14 packages (16,0 MB)
Downloading from 'fedora'... done
Importing packages... done
error: importing RPMs: package qt5-qtquickcontrols-5.15.13-1.fc40.x86_64 cannot be verified and repo fedora is GPG enabled: /var/cache/rpm-ostree/repomd/fedora-40-x86_64/packages/qt5-qtquickcontrols-5.15.13-1.fc40.x86_64.rpm could not be verified.
/var/cache/rpm-ostree/repomd/fedora-40-x86_64/packages/qt5-qtquickcontrols-5.15.13-1.fc40.x86_64.rpm:  PRÜFSUMME:  SIGNATURE:  NICHT OK

This has worked before (I am in a little cycle downgrading and upgrading due to testing things, so I wonder why I now get such a strange error for that one package. Also, strangely enough, it's also reproducible…)

$ ls -la /var/cache/rpm-ostree/repomd/fedora-40-x86_64/packages/qt5-*                                         
-rw-r--r--. 1 root root 1774195 30. Apr 00:32 /var/cache/rpm-ostree/repomd/fedora-40-x86_64/packages/qt5-qtquickcontrols2-5.15.13-1.fc40.x86_64.rpm
-rw-r--r--. 1 root root  463097 30. Apr 00:32 /var/cache/rpm-ostree/repomd/fedora-40-x86_64/packages/qt5-qtquickcontrols-5.15.13-1.fc40.x86_64.rpm
-rw-r--r--. 1 root root  101882 30. Apr 00:32 /var/cache/rpm-ostree/repomd/fedora-40-x86_64/packages/qt5-qtwebchannel-5.15.13-1.fc40.x86_64.rpm
-rw-r--r--. 1 root root 3351855 30. Apr 00:32 /var/cache/rpm-ostree/repomd/fedora-40-x86_64/packages/qt5-qtwebengine-5.15.16-4.fc40.x86_64.rpm
-rw-r--r--. 1 root root   89183 30. Apr 00:32 /var/cache/rpm-ostree/repomd/fedora-40-x86_64/packages/qt5-qtwebsockets-5.15.13-1.fc40.x86_64.rpm
$ sha256sum /var/cache/rpm-ostree/repomd/fedora-40-x86_64/packages/qt5-qtquickcontrols-5.15.13-1.fc40.x86_64.rpm        
9fdef8d117b407ff7fd220d64762ad550a6afe34aa5cb3a06eb69e3850ddbdd9  /var/cache/rpm-ostree/repomd/fedora-40-x86_64/packages/qt5-qtquickcontrols-5.15.13-1.fc40.x86_64.rpm

System details

$ rpm-ostree status -b                                
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
BootedDeployment:
● fedora:fedora/39/x86_64/silverblue
                  Version: 39.20240429.0 (2024-04-29T00:42:04Z)
               BaseCommit: 190f07d86271feae4848cb21c0bebc21572341b08365d36313b7052a54c1399e
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: ***

Additional information

The workaround is to delete the cache via rpm-ostree cleanup -bm, afterwards the rebase works successfully.

However, should that really be needed? Also, you can see I needed many many support requests and this one was one of it – as a usual user of rpm-ostree I cannot resolve the issue alone, as the error message does not indicate the error. So how should I know that cache clearing is a solution here?

Also if that cache clearing is needed and intentional, why does not -p (which, I needed to revert rpm-ostree reset) implicitly remove the cache etc.?