`--installroot` not compatible with some packages?
polarathene opened this issue · 17 comments
I noticed some packages had trouble installing properly with --installroot.
Often the error would not occur if I installed the package locally without --installroot, and recently I noticed ca-certificates would install with a symlink to it's content, but that content did not exist in the --installroot location where it should have been...?
zypper --releasever 15.4 --installroot /rootfs --gpg-auto-import-keys refresh
zypper --releasever 15.4 --installroot /rootfs --non-interactive install ca-certificates
ls /rootfs/etc/ssl/certs
# symlinked location when /rootfs becomes /, but no contents stored:
ls /rootfs/var/lib/ca-certificates/pem
ls /var/lib/ca-certificates/pemI have not looked into the package itself, but perhaps it tries to install the content via the symlink and that stores the content in the wrong location outside of /rootfs? (not verified).
For an example of a package that fails to install, busybox-adduser fails due to addgroup command failing:
Install log
zypper --releasever 15.4 --installroot /rootfs --gpg-auto-import-keys refresh && zypper --releasever 15.4 --installroot /rootfs --non-interactive install busybox-adduser
Warning: Enforced setting: $releasever=15.4
Repository 'Update repository of openSUSE Backports' is up to date.
Repository 'Non-OSS Repository' is up to date.
Repository 'Main Repository' is up to date.
Repository 'Update repository with updates from SUSE Linux Enterprise 15' is up to date.
Repository 'Main Update Repository' is up to date.
Repository 'Update Repository (Non-Oss)' is up to date.
All repositories have been refreshed.
Warning: Enforced setting: $releasever=15.4
Loading repository data...
Reading installed packages...
Resolving package dependencies...
The following 6 NEW packages are going to be installed:
busybox busybox-adduser libcrypt1 libsepol1 system-user-nobody sysuser-shadow
6 new packages to install.
Overall download size: 999.1 KiB. Already cached: 0 B. After the operation, additional 2.1 MiB will be used.
Continue? [y/n/v/...? shows all options] (y): y
Checking for file conflicts: (6 skipped) ......................................................................................................................................................................[done]
Warning: 6 packages had to be excluded from file conflicts check because they are not yet downloaded.
Note: Checking for file conflicts requires not installed packages to be downloaded in advance in
order to access their file lists. See option '--download-in-advance / --dry-run --download-only'
in the zypper manual page for details.
Retrieving package libsepol1-3.1-150400.1.70.x86_64 (1/6), 257.4 KiB (693.3 KiB unpacked)
Retrieving: libsepol1-3.1-150400.1.70.x86_64.rpm ..............................................................................................................................................................[done]
(1/6) Installing: libsepol1-3.1-150400.1.70.x86_64 ............................................................................................................................................................[done]
Retrieving package libcrypt1-4.4.15-150300.4.4.3.x86_64 (2/6), 104.7 KiB (259.0 KiB unpacked)
Retrieving: libcrypt1-4.4.15-150300.4.4.3.x86_64.rpm ..........................................................................................................................................................[done]
(2/6) Installing: libcrypt1-4.4.15-150300.4.4.3.x86_64 ........................................................................................................................................................[done]
Retrieving package busybox-1.35.0-150400.3.8.1.x86_64 (3/6), 601.4 KiB ( 1.2 MiB unpacked)
Retrieving: busybox-1.35.0-150400.3.8.1.x86_64.rpm ............................................................................................................................................................[done]
(3/6) Installing: busybox-1.35.0-150400.3.8.1.x86_64 ..........................................................................................................................................................[done]
Retrieving package sysuser-shadow-3.1-150400.1.35.noarch (4/6), 12.9 KiB ( 2.5 KiB unpacked)
Retrieving: sysuser-shadow-3.1-150400.1.35.noarch.rpm .........................................................................................................................................................[done]
(4/6) Installing: sysuser-shadow-3.1-150400.1.35.noarch .......................................................................................................................................................[done]
Retrieving package system-user-nobody-20170617-150400.22.33.noarch (5/6), 12.1 KiB ( 99 B unpacked)
Retrieving: system-user-nobody-20170617-150400.22.33.noarch.rpm ...............................................................................................................................................[done]
/usr/bin/busybox addgroup -S -g 65533 nogroup
addgroup: can't set default file creation context to system_u:object_r:container_ro_file_t:s0: No such file or directory
error: %prein(system-user-nobody-20170617-150400.22.33.noarch) scriptlet failed, exit status 1
error: system-user-nobody-20170617-150400.22.33.noarch: install failed
(5/6) Installing: system-user-nobody-20170617-150400.22.33.noarch ............................................................................................................................................[error]
Installation of system-user-nobody-20170617-150400.22.33.noarch failed:
Error: Subprocess failed. Error: RPM failed: Command exited with status 1.
Abort, retry, ignore? [a/r/i] (a): a
Problem occurred during or after installation or removal of packages:
Installation has been aborted as directed.
Please see the above error message for a hint.
Specifically:
/usr/bin/busybox addgroup -S -g 65533 nogroup
addgroup: can't set default file creation context to system_u:object_r:container_ro_file_t:s0: No such file or directory
error: %prein(system-user-nobody-20170617-150400.22.33.noarch) scriptlet failed, exit status 1
error: system-user-nobody-20170617-150400.22.33.noarch: install failed
Commands were run via Docker image opensuse/leap:15.4, with host OS Fedora 37 kernel 6.1.9, Docker Engine v23.
FWIW, dnf --installroot for the ca-certificates package also uses an absolute symlink, but still installs a copy of that data at the equivalent location in /rootfs so that it's carried over when /rootfs becomes / in a container image.
It's not a matter of ca-certificates or symlinks. If you look into ca-certificates you'll see that it installs the /etc/ssl/certs symlink pointing to /var/lib/ca-certificates/pem and contains /var/lib/ca-certificates/pem as an empty directory. It's correct that there are no data inside.
And I don't think this is directly related to --install-root either. As you wrote, the busybox addgroup command fails, in the %prein script of package system-user-nobody:
/usr/bin/busybox addgroup -S -g 65533 nogroup
addgroup: can't set default file creation context to system_u:object_r:container_ro_file_t:s0: No such file or directory
The error message mentions a SELinux label, so I'd assume the issue is somehow related to SELinux and/or dockers security options. But I'm not familiar with these. Better ask on the opensuse mailinglist.
If you look into ca-certificates you'll see that it installs the
/etc/ssl/certs symlinkpointing to/var/lib/ca-certificates/pemand contains/var/lib/ca-certificates/pemas an empty directory.
It's correct that there are no data inside.
I am rather new to using zypper, is this a command to provide that? Or is it from inspecting the package contents? (also something I'm not familiar with doing).
I had tried to find out what package provides the .pem files I was expecting when installing ca-certificates in other distros package managers, but was not having luck with commands like zypper search -f '*/Amazon_ROOT_CA_1.pem'. Eventually I figured I could try removing ca-certificates on the system and that mentioned ca-certificates-mozilla (which isn't suggested when installing ca-certificates), and that turned out to be the package with the expected contents.
Any advice on how to troubleshoot this sort of thing in the future would be appreciated :)
And I don't think this is directly related to
--install-rooteither. As you wrote, thebusybox addgroupcommand fails, in the%preinscript of packagesystem-user-nobody:
It was just an observation I noticed where I'd find packages failing to install with --install-root, sometimes not consistently and wasn't sure what else could be the cause.
The error message mentions a SELinux label
Probably because I tried this on a Fedora VM as the Docker host.
Thanks for pointing that out, I was not aware that was a failure due to SELinux, I assume system_u:object_r:container_ro_file_t:s0 is the label?
But I'm not familiar with these. Better ask on the opensuse mailinglist.
No worries, that was plenty helpful, thanks for clearing up the misunderstanding!
Any advice on how to troubleshoot this sort of thing in the future would be appreciated :)
Files installed on the system are IMO best queried directly via rpm. Query-file (-qf) tells you which package owns an exising file or directory on disk:
# rpm -qf /etc/ssl/certs
ca-certificates-2+git20170807.10b2785-lp152.8.2.noarch
The content of an installed package can be listed with -l or -lv:
# rpm -qf /etc/ssl/certs -lv
# rpm -q ca-certificates -lv
For files available in not installed packages provided by your repos, we currently have no solution. The filelists of all packages included in a repo is very large. That's why we don't download it when refreshing the repos. zypper search -f would be the right command, but we need to implement some on-demand download of the filelists first. Without this it won't find files in not installed packages.
Run on a host without SELinux, and on the Fedora SELinux host with docker run --rm -it --privileged opensuse/leap:15.4 bash. The busybox-adduser is non-issue, but I've noticed the following problem is happening.
ca-certificates is not populating the content that it should, because the update-ca-certificates script can't access /dev/fd/62. I've seen other packages also complain about no access to /dev/null etc.
$ zypper --non-interactive --gpg-auto-import-keys --releasever 15.4 --installroot /rootfs \
install ca-certificates-mozilla
# ...
(33/35) Installing: openssl-1_1-1.1.1l-150400.7.28.1.x86_64 ...................................[done]
Retrieving package ca-certificates-2+git20210309.21162a6-2.1.noarch
(34/35), 27.7 KiB ( 26.7 KiB unpacked)
Retrieving: ca-certificates-2+git20210309.21162a6-2.1.noarch.rpm ..............................[done]
/usr/sbin/update-ca-certificates: line 71: /dev/fd/62: No such file or directory
sort: fflush failed: 'standard output': Broken pipe
sort: write error
(34/35) Installing: ca-certificates-2+git20210309.21162a6-2.1.noarch ..........................[done]
Retrieving package ca-certificates-mozilla-2.60-150200.27.1.noarch
(35/35), 364.4 KiB (967.7 KiB unpacked)
Retrieving: ca-certificates-mozilla-2.60-150200.27.1.noarch.rpm ...............................[done]
/usr/sbin/update-ca-certificates: line 71: /dev/fd/62: No such file or directory
sort: fflush failed: 'standard output': Broken pipe
sort: write error
(35/35) Installing: ca-certificates-mozilla-2.60-150200.27.1.noarch ...........................[done]
/usr/sbin/update-ca-certificates: line 71: /dev/fd/62: No such file or directory
Executing %posttrans scripts ..................................................................[done]Note the failures to run /usr/sbin/update-ca-certificates.
# No file content:
$ ls -lah /rootfs/var/lib/ca-certificates/pem
total 8.0K
dr-xr-xr-x. 2 root root 4.0K Apr 8 2021 .
drwxr-xr-x. 4 root root 4.0K Apr 4 00:22 ..
# Main root has contents:
$ ls -lah /var/lib/ca-certificates/pem
# ...
lrwxrwxrwx. 1 root root 20 Mar 13 10:13 fd08c599.0 -> Amazon_Root_CA_1.pem
# ...
-r--r--r--. 1 root root 1.9K Mar 13 10:13 vTrus_Root_CA.pem
# Folder belongs to `ca-certificates` package:
$ rpm -qf /var/lib/ca-certificates/pem
ca-certificates-2+git20210309.21162a6-2.1.noarch
$ rpm --root /rootfs -qf /var/lib/ca-certificates/pem
ca-certificates-2+git20210309.21162a6-2.1.noarch
$ rpm --root /rootfs -qf /var/lib/ca-certificates/pem -lv
# ...
dr-xr-xr-x 2 root root 0 Apr 8 2021 /var/lib/ca-certificates/pem
# Query package for file content inside location:
$ rpm -qf /var/lib/ca-certificates/pem/vTrus_Root_CA.pem
file /var/lib/ca-certificates/pem/vTrus_Root_CA.pem is not owned by any package
# Not generated due failure from `/usr/sbin/update-ca-certificates`:
$ rpm -r /rootfs -qf /usr/sbin/update-ca-certificates
ca-certificates-2+git20210309.21162a6-2.1.noarch
$ chroot /rootfs /usr/sbin/update-ca-certificates -v -f
/usr/sbin/update-ca-certificates: line 71: /dev/fd/62: No such file or directory
$ /usr/sbin/update-ca-certificates -v -f
running /usr/lib/ca-certificates/update.d/50java.run ..
creating /var/lib/ca-certificates/java-cacerts ...
running /usr/lib/ca-certificates/update.d/70openssl.run ..
creating /var/lib/ca-certificates/openssl ...
running /usr/lib/ca-certificates/update.d/80etc_ssl.run ..
running /usr/lib/ca-certificates/update.d/99certbundle.run ..
creating /var/lib/ca-certificates/ca-bundle.pem ...To verify any potential confusion with resolving from the --installroot location, here's the equivalent with the ripgrep package only installed into /rootfs:
$ zypper --non-interactive --gpg-auto-import-keys --releasever 15.4 --installroot /rootfs \
install ripgrep
$ rpm -qf /usr/bin/rg
error: file /usr/bin/rg: No such file or directory
$ rpm --root /rootfs -qf /usr/bin/rg
ripgrep-13.0.0-bp154.1.26.x86_64Is this a compatibility issue? It does work once I run a container that uses /rootfs as /. I am not sure how the runtime locations like /dev are handled, but I'm not sure if this would be that different from Docker vs a VM or bare-metal install?
I hope @lnussel is able to help us with this. Mabe the script (package providing it) lacks some dependencies.
No dependency issue, it looks more like the package requires /dev at install-time. So it can not be installed using a plain rpm --root call (which is what zypper does). Maybe we can work around this ins ZYPP, and temp. mount /dev in the installroot.
@polarathene a workaround until the enhancement is released (libzypp-17.31.10) could be to manually mount and unmount /dev into the chroot at /rootfs:
mount --rbind --make-rslave /dev /rootfs/dev
If this succeeded, the above --installroot /rootfs commands should find the expected files below (/rootfs)/dev. If all commands are done, you must unmount it again:
umount -R /dev /rootfs/dev
mount --rbind --make-rslave /dev /rootfs/dev
Is this what you meant?
$ docker run --rm -it opensuse/leap:15.4
$ mkdir -p /rootfs/dev
$ mount --rbind --make-rslave /dev /rootfs/dev
mount: /rootfs/dev: permission denied.Host is Arch Linux, also tried on Fedora with SELinux set to permissive mode. Both running Docker Engine v23 with container user as root.
I assume it's not permitted by Docker by default, and I just need to allow/add some capability. If that would be required regardless I understand, but it makes zypper less desirable in a Dockerfile for users to build an image from.
Not sure how it differs from Fedora's package with dnf --installroot which does work fine. Perhaps the package could take a different approach to avoid relying on /dev if viable.
In this case, I think I can just copy the same content that ca-certificates would generate from the main root to /rootfs location, or once I COPY the /rootfs into the image builds next FROM scratch stage to / and run the /usr/sbin/update-ca-certificates from there.
I just wanted to raise awareness about it, may affect other packages 👍
The error is from a process substitution in a shell script. That could happen with any package. There's just some basic /dev infrastructure that has to always exist, such as /dev/null(!) or /dev/fd. The latter is a symlink to /proc/self/fd, ie /proc needs to be mounted too. Also one likely wants /run as tmpfs and probably a few other API like fs infrastructure. See systemd-nspawn for some logic.
But that's something zypper can not control. Neither the packages not the permission inside the container. The brute force approach would be to run the container without any security (--privileged). In this case forwarding /proc and /dev most probably succeeds. Otherwise it's a matter of finding the offending security check and adapt the containers profile.
Providing a /dev/null would be doable, but for /proc zypp needs permission to mount it.
zypp runs as root. can do whatever it wants :-) Whether or not setting up this basic infra is zypp's job is another topic :-)
Not inside a container:
ma@hobbes:~ (0)> docker run --rm -ti registry.opensuse.org/opensuse/leap:15.4
b6d89af8778a:/ # id
uid=0(root) gid=0(root) groups=0(root)
b6d89af8778a:/ # mkdir -p /rootfs/dev
b6d89af8778a:/ # mount --rbind --make-rslave /dev /rootfs/dev
mount: /rootfs/dev: permission denied.
b6d89af8778a:/ #
You need to relax the security options
ma@hobbes:~ (130)> docker run --rm -ti --privileged registry.opensuse.org/opensuse/leap:15.4
8e87f4b6400c:/ # id
uid=0(root) gid=0(root) groups=0(root)
8e87f4b6400c:/ # mkdir -p /rootfs/dev
8e87f4b6400c:/ # mount --rbind --make-rslave /dev /rootfs/dev
8e87f4b6400c:/ #
not sure how that privilege system works there. In general you'd need to have CAP_SYS_ADMIN within the user namespace of a container.
The error is from a process substitution in a shell script. That could happen with any package.
There's just some basic/devinfrastructure that has to always exist, such as/dev/null(!) or/dev/fd.
This is probably a package that could be reworked to avoid that incompatibility with --installroot? Doesn't look like Fedora's ca-certificates package relies on anything from /dev to install, although they don't seem to have a update-ca-certificates script equivalent either from the looks of it?
I tried with the Fedora postfix package and dnf --installroot also didn't output any errors despite all the /dev/null in the install/postinstall scripts of the linked .spec.
I assume that means dnf is avoiding that issue by doing things differently?
A simple redirect >/dev/null will work without error as soon as a /dev directory exists. If no devtmpfs is mounted, it will simply cause a file /dev/null to be created. If the system later mounts devtmpfs to /dev, the file will be shadowed. This just wastes some disk space and will probably never be noticed.
AFAICS dnf does not deal much with /dev or /proc, so I suppose the Fedora packages are simply less demanding in this regard.