An error occurs when rebasing on a layered OCI image
Closed this issue · 24 comments
fengz63 commented
An image built from a Dockerfile like:
FROM localhost:5000/my-os:latest
RUN rpm-ostree install wgetThat can be used to run a container successfully. However, when using it for rebasing we get a fatal error:
$sudo rpm-ostree rebase --experimental ostree-unverified-registry:localhost:5000/my-os-custom:latest
Pulling manifest: ostree-unverified-image:docker://localhost:5000/my-os-custom:latest
Importing: ostree-unverified-image:docker://localhost:5000/my-os-custom:latest (digest: sha256:2341613f4ef1be4085359b4617997877a77735359ec51c2b37da08fb68a52758)
ostree chunk layers stored: 1 needed: 0 (0 bytes)
custom layers stored: 0 needed: 1 (11.0 MB)
Fetching layer sha256:7c5cc0027a45 (11.0 MB)
error: Parsing layer blob sha256:7c5cc0027a450e52200a8fdc59c48b43b281d87191b7e79b07e5fc7c6e615344: Failed to commit tar: ExitStatus(unix_wait_status(256)): error: ostree-tar: Processing deferred hardlink sysroot/ostree/repo/objects/72/53f994e6f4a1fd5cce059fe4654f08cbd7cc80a2b8472f959514dd768fa381.file: Failed to find object: No such file or directory: sysrootSo, does anyone has the same problem?
lucab commented
Thanks for the report, but you completely skipped the template so this ticket does not have enough details to debug.
I think this may be some version incompatibility or some mixup in my-os:latest or my-os-custom:latest.
fengz63 commented
The my-os:latest is my own os based on ostree. my-os-custom is generated by the Dockerfile listed above.
And I have found a strange phenomenon. When I install some packages, for example curl, the Dockerfile is as follows:
FROM localhost:5000/my-os:latest
RUN rpm-ostree install curlthen, I could rebase to the new OCI image built by the dockerfile successfully. But with some packages, I couldn't rebase to the custom image.
cgwalters commented
Which version of rpm-ostree are you using?
fengz63 commented
The rpm-ostree version is 2022.8 and the ostree version is 2022.2.
fengz63 commented
Dose anyone has a suggestion for the above questions?
cgwalters commented
If you can minimize this as an example against e.g. the current FCOS image that would help. I suspect you have an older ostree/rpm-ostree in the base image.
For example, using
FROM quay.io/coreos-assembler/fcos:stable
RUN rpm-ostree install wget
I don't see this error. However note - you should be running through the clean/commit process too! See e.g. this example https://github.com/coreos/coreos-layering-examples/blob/90ee691f496325232d326f27a5a1bb4c3203b5bd/rsyslog/Dockerfile#L3
Specifically this should really look like:
FROM quay.io/coreos-assembler/fcos:stable
RUN rpm-ostree install wget && \
rpm-ostree cleanup -m && \
ostree container commit
for example.
cgwalters commented
Tentatively closing for lack of information. Please do feel free to reopen if you've managed to debug this further!
cgwalters commented
Reopening since we seem to be narrowing in on a reproducer in jmarrero/kinoite-setup#1
cgwalters commented
(This seems likely to really be an ostree-ext issue, but we can transfer the issue when we narrow it down more precisely)
Hmm one thing I notice here in that test case data is:
$ tar tvf ./img/blobs/sha256/82cffdd9305f339b0b490212f261c594ac96b1b13cd76cb7f795def9e208047b |grep sysroot
drwxr-xr-x root/root 0 1969-12-31 19:00 sysroot/
drwxr-xr-x root/root 0 1969-12-31 19:00 sysroot/ostree/
drwxr-xr-x root/root 0 1969-12-31 19:00 sysroot/ostree/repo/
drwxr-xr-x root/root 0 1969-12-31 19:00 sysroot/ostree/repo/objects/
drwxr-xr-x root/root 0 1969-12-31 19:00 sysroot/ostree/repo/objects/ae/
hrw-r--r-- root/root 0 2022-11-07 17:00 sysroot/ostree/repo/objects/ae/82cbc53364b8d4f1a673ea9a26a20ddb427c4f75e7a4968eb1592b726ebd25.file link to etc/shells
drwxr-xr-x root/root 0 1969-12-31 19:00 sysroot/ostree/repo/objects/bf/
-rw-r--r-- root/root 98304 2022-11-07 17:00 sysroot/ostree/repo/objects/bf/13d8fe63612425b89f526a906f06359a1a71ac71902caef8a7160558d91187.file
drwxr-xr-x root/root 0 1969-12-31 19:00 sysroot/ostree/repo/objects/db/
-rw-r--r-- root/root 58208256 2022-11-07 17:00 sysroot/ostree/repo/objects/db/94aa5f8f0e502d3777575b9137af3aef896a60b638bee3da7e13e230f729f9.file
drwxr-xr-x root/root 0 1969-12-31 19:00 sysroot/ostree/repo/objects/ea/
-rw-r--r-- root/root 0 2022-11-07 17:00 sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite-wal link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/alsa/ucm2/conf.virt.d/.gitignore link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/crypto-policies/EMPTY/libssh.txt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/doc/NetworkManager-ssh/ChangeLog link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/doc/apr/README.deepbind.deepbind link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/doc/fedora-workstation-backgrounds/README link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/rpm/rpmdb.sqlite-wal link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/xml/iso-codes/iso_3166-3.xml link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
Basically we have hardlinks into /sysroot/ostree in a non-ostree layer. I wonder if different container build systems are going to be handling the hardlinks here differently - potentially it's ordering dependent or even racy?
I don't see any hardlinks to /sysroot in the images I've built.
cgwalters commented
@jmarrero so what I've been doing is using e.g. skopeo copy docker://quay.io/myimage oci:ocidir:myimage and then inspecting the raw tar stream in the layers via e.g. tar tvf blobs/sha256/<digest>.
Can you try to diff a successful and failing image in that way?
jmarrero commented
No errors with: quay.io/jmarrero_rh/my-custom-fcos:silverblue-2
But:
skopeo copy docker://quay.io/jmarrero_rh/my-custom-fcos:silverblue-2 oci:ocidir-good:good-image
sha256 tar tvf ff5f4ae9d98fb4869f5bb5ea5e749ccc6f068f7db5b8e7f3505cd0f2262c65ba |grep sysroot
drwxr-xr-x 0/0 0 1969-12-31 19:00 sysroot
drwxr-xr-x 0/0 0 1969-12-31 19:00 sysroot/ostree
...
drwxr-xr-x 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/ff
drwxr-xr-x 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/extensions
drwxr-xr-x 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/refs
drwxr-xr-x 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/refs/heads
drwxr-xr-x 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/refs/mirrors
drwxr-xr-x 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/refs/remotes
drwxr-xr-x 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/state
drwxr-xr-x 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/tmp
drwxr-xr-x 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/tmp/cache
-rw-r--r-- 0/0 45 1969-12-31 19:00 sysroot/ostree/repo/config
-rw-r--r-- 0/0 46 1969-12-31 19:00 sysroot/ostree/repo/objects/ea/ca5cbabc63c3dc577b6bd4e21f169fcd649f5ff50c037e2f1a90840e73a24d.file-xattrs
hrw-r--r-- 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/45/8d2ed1ec76eac1a2bed1ecc6b16a4a1b78e96b054b48f84d13762e60b96efa.file-xattrs-link link to sysroot/ostree/repo/objects/ea/ca5cbabc63c3dc577b6bd4e21f169fcd649f5ff50c037e2f1a90840e73a24d.file-xattrs
lrwxrwxrwx 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/45/8d2ed1ec76eac1a2bed1ecc6b16a4a1b78e96b054b48f84d13762e60b96efa.file -> ../../../../usr/lib64/libmozjs-102.so.0.0.0
hrwxrwxrwx 0/0 0 1969-12-31 19:00 usr/lib/.build-id/09/841a58225c4f81f430f5f25d1d885a17fdc03f link to sysroot/ostree/repo/objects/45/8d2ed1ec76eac1a2bed1ecc6b16a4a1b78e96b054b48f84d13762e60b96efa.file
-rw-r--r-- 0/0 46 1969-12-31 19:00 sysroot/ostree/repo/objects/98/223daccac009b54901f56d64902ae3c54f248baea8bf9da4d35a4a69006e3a.file-xattrs
hrw-r--r-- 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/5b/585ca239cf94cb9a33bb1f7b7e29119b17d7937c8212a73f9bb1a4f715acb3.file-xattrs-link link to sysroot/ostree/repo/objects/98/223daccac009b54901f56d64902ae3c54f248baea8bf9da4d35a4a69006e3a.file-xattrs
-rw-r--r-- 0/0 696 1969-12-31 19:00 sysroot/ostree/repo/objects/5b/585ca239cf94cb9a33bb1f7b7e29119b17d7937c8212a73f9bb1a4f715acb3.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/doc/mozjs102/README.html link to sysroot/ostree/repo/objects/5b/585ca239cf94cb9a33bb1f7b7e29119b17d7937c8212a73f9bb1a4f715acb3.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/78/1805fc17b80140e08a0fb7114630ddbca97ee935068974b387969c5bd011eb.file-xattrs-link link to sysroot/ostree/repo/objects/ea/ca5cbabc63c3dc577b6bd4e21f169fcd649f5ff50c037e2f1a90840e73a24d.file-xattrs
-rwxr-xr-x 0/0 13411408 1969-12-31 19:00 sysroot/ostree/repo/objects/78/1805fc17b80140e08a0fb7114630ddbca97ee935068974b387969c5bd011eb.file
hrwxr-xr-x 0/0 0 1969-12-31 19:00 usr/lib64/libmozjs-102.so.0.0.0 link to sysroot/ostree/repo/objects/78/1805fc17b80140e08a0fb7114630ddbca97ee935068974b387969c5bd011eb.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/98/679c545e63b1270fd33c3d6854eb9d7a3e9f8544ff0e31c8d43deb93fb00f3.file-xattrs-link link to sysroot/ostree/repo/objects/ea/ca5cbabc63c3dc577b6bd4e21f169fcd649f5ff50c037e2f1a90840e73a24d.file-xattrs
lrwxrwxrwx 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/98/679c545e63b1270fd33c3d6854eb9d7a3e9f8544ff0e31c8d43deb93fb00f3.file -> libmozjs-102.so.0.0.0
hrwxrwxrwx 0/0 0 1969-12-31 19:00 usr/lib64/libmozjs-102.so.0 link to sysroot/ostree/repo/objects/98/679c545e63b1270fd33c3d6854eb9d7a3e9f8544ff0e31c8d43deb93fb00f3.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/b5/d1870e6c304abf6e4eae7b79db15b1695fbbaa7614c0cdbd8918a1f2b11ca2.file-xattrs-link link to sysroot/ostree/repo/objects/98/223daccac009b54901f56d64902ae3c54f248baea8bf9da4d35a4a69006e3a.file-xattrs
-rw-r--r-- 0/0 389 1969-12-31 19:00 sysroot/ostree/repo/objects/b5/d1870e6c304abf6e4eae7b79db15b1695fbbaa7614c0cdbd8918a1f2b11ca2.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/licenses/mozjs102/LICENSE link to sysroot/ostree/repo/objects/b5/d1870e6c304abf6e4eae7b79db15b1695fbbaa7614c0cdbd8918a1f2b11ca2.file
With
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/doc/mozjs102/README.html link to sysroot/ostree/repo/objects/5b/585ca239cf94cb9a33bb1f7b7e29119b17d7937c8212a73f9bb1a4f715acb3.file
Is what we are looking for right? which appear to exist on both my good and bad builds. I will find the exact files hardlinking to the failing object and compare those.
jmarrero commented
I think this might be related...
Bad image:
./sha256/7cc83e20a3824c08397341669c9ecf07780cc68aaf6352e1692c704d74b5d25d
lrwxrwxrwx 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/5d/838fcee94ad472ea96dbc718a965e0674f751b3eb90fb140dd72ad9d0eff06.file -> /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
./sha256/82cffdd9305f339b0b490212f261c594ac96b1b13cd76cb7f795def9e208047b
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
good image:
./sha256/fb6fc1e2e97291ad39dbe53434e19cb244d9eceeedc07a0c0b094cf0cc7aefc0
lrwxrwxrwx 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/5d/838fcee94ad472ea96dbc718a965e0674f751b3eb90fb140dd72ad9d0eff06.file -> /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
This while running:
for filename in ./sha256/*
do
echo "${filename}" && tar tvf "${filename}" |grep "usr/share/pki/ca-trust-legacy/ca-bundle"
done
cgwalters commented
In our base images, basically all regular files are hardlinked. So modifying files as part of the base image will usually actually modify both "names" for the same file.
When generating a tar stream...I think it's up to the building process which way to create the links. It may be in some cases it varies.
Our current filtering code will basically drop on the floor anything in /sysroot, so that's why we're seeing the error.
But wait a second...I think something may be going on here specifically with zero-sized files. I notice this in your build:
tar tvf blobs/sha256/82cffdd9305f339b0b490212f261c594ac96b1b13cd76cb7f795def9e208047b |grep objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e
-rw-r--r-- root/root 0 2022-11-07 17:00 sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite-wal link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/alsa/ucm2/conf.virt.d/.gitignore link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/crypto-policies/EMPTY/libssh.txt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/doc/NetworkManager-ssh/ChangeLog link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/doc/apr/README.deepbind.deepbind link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/doc/fedora-workstation-backgrounds/README link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/rpm/rpmdb.sqlite-wal link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/xml/iso-codes/iso_3166-3.xml link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
Notice how many hardlinks there are to the empty file. I bet this has something to do with it. We should not be hardlinking empty files.
jmarrero commented
Well but in the good build I still have the same behavior:
./sha256/fb6fc1e2e97291ad39dbe53434e19cb244d9eceeedc07a0c0b094cf0cc7aefc0
hrw-r--r-- 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file-xattrs-link link to sysroot/ostree/repo/objects/98/223daccac009b54901f56d64902ae3c54f248baea8bf9da4d35a4a69006e3a.file-xattrs
-rw-r--r-- 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite-wal link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/alsa/ucm2/conf.virt.d/.gitignore link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/crypto-policies/EMPTY/libssh.txt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/doc/NetworkManager-ssh/ChangeLog link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/doc/apr/README.deepbind.deepbind link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/doc/fedora-workstation-backgrounds/README link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/mime/icons link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/rpm/rpmdb.sqlite-wal link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/xml/iso-codes/iso_3166-3.xml link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
The difference I see with the bad one is still that they exists in two tars and not just one.
./sha256/7cc83e20a3824c08397341669c9ecf07780cc68aaf6352e1692c704d74b5d25d
hrw-r--r-- 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file-xattrs-link link to sysroot/ostree/repo/objects/98/223daccac009b54901f56d64902ae3c54f248baea8bf9da4d35a4a69006e3a.file-xattrs
-rw-r--r-- 0/0 0 1969-12-31 19:00 sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite-wal link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/alsa/ucm2/conf.virt.d/.gitignore link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/crypto-policies/EMPTY/libssh.txt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/doc/NetworkManager-ssh/ChangeLog link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/doc/apr/README.deepbind.deepbind link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/doc/fedora-workstation-backgrounds/README link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/mime/icons link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/rpm/rpmdb.sqlite-wal link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- 0/0 0 1969-12-31 19:00 usr/share/xml/iso-codes/iso_3166-3.xml link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
and
./sha256/82cffdd9305f339b0b490212f261c594ac96b1b13cd76cb7f795def9e208047b
-rw-r--r-- root/root 0 2022-11-07 17:00 sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite-wal link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/alsa/ucm2/conf.virt.d/.gitignore link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/crypto-policies/EMPTY/libssh.txt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/doc/NetworkManager-ssh/ChangeLog link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/doc/apr/README.deepbind.deepbind link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/doc/fedora-workstation-backgrounds/README link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/rpm/rpmdb.sqlite-wal link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-07 17:00 usr/share/xml/iso-codes/iso_3166-3.xml link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
I am searching using:
for filename in ./sha256/*
do
echo "${filename}" && tar tvf "${filename}" |grep "6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e"
done
The other weird thing is that on the good build, the timestamp are 1969 but the bad build the second tar has the same files with the timestamp of the actual moment I did the build.
cgwalters commented
I've minimized it to this:
FROM quay.io/fedora/fedora-coreos:stable
RUN touch /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt
Building that using the redhat-actions/buildah-build@v2 in GH Actions, then rebasing causes the failure. Looking at the generated tarball from that single touch:
drwxr-xr-x root/root 0 2022-11-08 22:36 run/
-rwx------ root/root 0 2022-11-08 22:36 run/.containerenv
drwxr-xr-x root/root 0 2022-11-08 22:36 run/systemd/
drwxr-xr-x root/root 0 2022-11-08 22:36 run/systemd/resolve/
-rwx------ root/root 0 2022-11-08 22:36 run/systemd/resolve/stub-resolv.conf
drwxr-xr-x root/root 0 1970-01-01 00:00 sysroot/
drwxr-xr-x root/root 0 1970-01-01 00:00 sysroot/ostree/
drwxr-xr-x root/root 0 1970-01-01 00:00 sysroot/ostree/repo/
drwxr-xr-x root/root 0 1970-01-01 00:00 sysroot/ostree/repo/objects/
drwxr-xr-x root/root 0 1970-01-01 00:00 sysroot/ostree/repo/objects/ea/
-rw-r--r-- root/root 0 2022-11-08 22:36 sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
drwxr-xr-x root/root 0 2022-11-08 22:36 usr/
drwxr-xr-x root/root 0 2022-11-08 22:36 usr/lib/
drwxr-xr-x root/root 0 2022-11-08 22:36 usr/lib/sysimage/
drwxr-xr-x root/root 0 2022-11-08 22:36 usr/lib/sysimage/rpm-ostree-base-db/
hrw-r--r-- root/root 0 2022-11-08 22:36 usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite-wal link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
drwxr-xr-x root/root 0 2022-11-08 22:36 usr/share/
drwxr-xr-x root/root 0 2022-11-08 22:36 usr/share/crypto-policies/
drwxr-xr-x root/root 0 2022-11-08 22:36 usr/share/crypto-policies/EMPTY/
hrw-r--r-- root/root 0 2022-11-08 22:36 usr/share/crypto-policies/EMPTY/libssh.txt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
drwxr-xr-x root/root 0 2022-11-08 22:36 usr/share/mime/
hrw-r--r-- root/root 0 2022-11-08 22:36 usr/share/mime/icons link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
drwxr-xr-x root/root 0 2022-11-08 22:36 usr/share/pki/
drwxr-xr-x root/root 0 2022-11-08 22:36 usr/share/pki/ca-trust-legacy/
hrw-r--r-- root/root 0 2022-11-08 22:36 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
hrw-r--r-- root/root 0 2022-11-08 22:36 usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
drwxr-xr-x root/root 0 2022-11-08 22:36 usr/share/rpm/
hrw-r--r-- root/root 0 2022-11-08 22:36 usr/share/rpm/rpmdb.sqlite-wal link to sysroot/ostree/repo/objects/ea/6b004bc150b3b37d623348c7b77b3486d388f64c3f1138a021b8140d267d0e.file
cgwalters commented
OK I did ostreedev/ostree-rs-ext#400 and then I pushed a new FCOS base image up at quay.io/cgwalters/fcos:latest - and the same derivation test now works.
As I say in the PR, I don't consider that a complete fix, but it definitely helps.
cgwalters commented
OK yeah, this isn't just about empty files. I filed ostreedev/ostree-rs-ext#405 to track the necessary work on the ostree side to handle this.
Let's keep this open to track vendoring in a new ostree-rs-ext in this project.
cgwalters commented
This one should be fixed by #4140
Please feel free to try out the latest code (e.g. https://copr.fedorainfracloud.org/coprs/g/CoreOS/continuous/ ) to verify!
jmarrero commented
https://github.com/coreos/rpm-ostree/releases/tag/v2022.16 includes the fix and I verified it works now with my silverblue setup as well. Thank you Colin!
cgwalters commented
Awesome!
jmarrero commented
Well, I should have checked better. Today I noticed I was still pulling from the image built with docker and not the podman one and noticed this:
~ rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
error: Child process exited with code 1
check `journalctl -b -1 -u ostree-finalize-staged.service`
Nov 28 18:29:44 silverblue systemd[1]: Finished ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Nov 28 18:30:15 silverblue systemd[1]: Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
Nov 28 18:30:15 silverblue ostree[5106]: Finalizing staged deployment
Nov 28 18:30:16 silverblue ostree[5106]: Copying /etc changes: 26 modified, 1 removed, 144 added
Nov 28 18:30:16 silverblue ostree[5106]: Copying /etc changes: 26 modified, 1 removed, 144 added
Nov 28 18:30:16 silverblue ostree[5115]: The --rebuild-if-modules-changed option is deprecated. Use --refresh instead.
Nov 28 18:30:17 silverblue ostree[5115]: libsemanage.semanage_direct_get_module_info: Unable to open bumblebee module lang ext file at /etc/selinux/targeted/tmp/modules/100/bumblebee/lang_ext. (No such file or directory).
Nov 28 18:30:17 silverblue ostree[5115]: semodule: Failed!
Nov 28 18:30:17 silverblue ostree[5106]: error: Child process exited with code 1
Nov 28 18:30:17 silverblue systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
Nov 28 18:30:17 silverblue systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.
Nov 28 18:30:17 silverblue systemd[1]: Stopped ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Nov 28 18:30:17 silverblue systemd[1]: ostree-finalize-staged.service: Consumed 1.145s CPU time.
jmarrero commented
Ignore my last comment, I am guessing I had some weird combination of packages between the images. I reset my install and rebased on the supposedly bad image. And it just works now. I bet it was some rpm-ostree version mismatch between the old image and installed one.
cgwalters commented
Mmm I saw this bug too, I think it is a different issue than the one here. We should probably file it separately if we see it more but I think basically it's going to be masked by #4122 being used at build time.
jmarrero commented
Makes sense, if I see it again I will create a new bug.