Toolbox image build is failing

Question

Toolbox image build is failing

phlogistonjohn opened this issue 2 years ago · 9 comments

I also suspect a chicken-and-egg problem is preventing this from resolving naturally.

The error:

 Total                                           5.7 MB/s |  34 MB     00:05     
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction test error:
  file /usr/bin/systemd-tmpfiles from install of systemd-250.3-8.fc36.x86_64 conflicts with file from package systemd-standalone-tmpfiles-250.9-1.fc36.x86_64

The command '/bin/sh -c dnf -y install samba-test' returned a non-zero code: 1
make: *** [Makefile:138: .build.toolbox] Error 1

I was unable to reproduce this locally which leads me to thinking there's a chicken and egg problem. What I suspect is that at some point the fedora base image and the systemd update conflicted in the image that was created as samba-server:latest. Now every time it builds it builds based on the samba-server:latest on quay.io which has the problem. Then the build fails when the build fails a newer samba-server:latest is not pused to quay because all jobs must succeed to trigger the push.

I see two options:

We do not require all jobs to run for push. Each "chain" of actions relates to one image
During the build the job(s) responsible for building the toolbox share/reuse the samba-server image that was just built. This may mean combining the steps responsible for building those images.

I prefer item 2 above. It may seem a messy on first glance, but it reflects an underlying relationship between those images.

Answer 1 · 2023-01-17T20:53:05.000Z

cc: @synarete

Answer 2 · 2023-01-18T07:42:47.000Z

@phlogistonjohn Currently, the toolbox image is based on samba-client image.

IMHO (following you analysis), a solution could be to require that the make target build-toolbox will depend on build-client target, so it will always use a locally built image. I will submit a PR.

Answer 3 · 2023-01-18T09:58:27.000Z

I was able to reproduce this on my local machine (without access to quay.io), but I am not sure what is the root cause of this issue. Even when adding dnf clean packages and dnf clean all, it still fails with same error.

When building samba-toolbox directly from fedora:36 there isn't any problem. I think it may be best to fallback to this mode.

Answer 4 · 2023-01-18T11:03:50.000Z

After some careful reading I think I have an explanation to why this issue occur. I'll try to post it later.

Answer 5 · 2023-01-18T15:03:12.000Z

I was able to reproduce this on my local machine (without access to quay.io), but I am not sure what is the root cause of this issue. Even when adding dnf clean packages and dnf clean all, it still fails with same error.

When building samba-toolbox directly from fedora:36 there isn't any problem. I think it may be best to fallback to this mode.

You were able to reprocdue the systemd packaging error? Can you try pulling the fedora base image and then rebuilding all images too? I tried to "nuke" all my cached stuff before reproducing because podman/docker cache stuff so aggressively.

Answer 6 · 2023-01-18T17:42:28.000Z

After some careful reading I think I have an explanation to why this issue occur. I'll try to post it later.

TL;DR systemd-standalone-tmpfiles is a sub-package of systemd. It is defined as a preferred requirement within samba. On the other side it is purposefully meant to conflict with systemd of same version or higher.

Let's start with systemd-standalone-tmpfiles. It is an optional requirement for samba-common(pulled in by samba-client) defined as below in samba.spec:

Requires(post): (systemd-standalone-tmpfiles or systemd)

and systemd is a requirement for base samba package(pulled in ultimately by samba-test)

Requires(post): systemd
Requires(preun): systemd
Requires(postun): systemd

At the same time systemd.spec has the following conflict defined in a weird way:

Conflicts: %{name}-standalone-tmpfiles < %{version}-%{release}^

This means that systemd-standalone-tmpfiles can only be marked for install if and only if its version is higher than installed systemd. fedora:36 base image comes with a particular version of systemd-libs installed and till 2 weeks back(Samba Container Image CI was running fine then) there were no updated packages. Around that time our samba-client container build process used to install systemd instead of systemd-standalone-tmpfiles and subsequent samba-toolbox build could succeed with already installed systemd.

With newer/updated packages(for systemd) we started installing systemd-standalone-tmpfiles instead of systemd while building samba-client container. Following that samba-toolbox container build process tries to install systemd resulting in a conflict because same binary(/usr/bin/systemd-tmpfiles) is already installed via systemd-standalone-tmpfiles.

Due to above mentioned reasons installing both samba-test and samba-client packages directly within fedora:36 base image will resolve all dependencies by pulling systemd(and not systemd-standalone-tmpfiles) and related packages.

Answer 7 · 2023-01-18T18:31:24.000Z

Thanks! I hope its not to harsh of me to suggest this is really frustrating and difficult to understand on part of samba/systemd packages.
I guess that means having the containers no longer be layered is the easiest way out but it makes me a bit sad to see.

Answer 8 · 2023-01-19T02:48:26.000Z

Thanks! I hope its not to harsh of me to suggest this is really frustrating and difficult to understand on part of samba/systemd packages.

I can only agree to it.

In my perspective the way in which systemd handled this package split to contain conflicting files seems to be wrong. Fedora guidelines clearly states that the new version of the original package should be updated to not contain the conflicting files and to depend on the new package. Anyway its not something we should wait for to get fixed.

I guess that means having the containers no longer be layered is the easiest way out but it makes me a bit sad to see.

There is yet another possibility to bring back the original behaviour(of layering) and that is by explicitly installing systemd along with samba-client in samba-client container image build process. This will ensure that systemd-standalone-tmpfiles is never getting installed which is logically correct as it is meant for use in non-systemd systems(as per systemd.spec).

Finally, not to forget that this whole issue will disappear when fedora:36 base image is rebuilt with updated/current systemd available with stable repositories(until next update comes in 😐).

Answer 9 · 2023-01-19T07:15:58.000Z

After some careful reading I think I have an explanation to why this issue occur. I'll try to post it later.

TL;DR systemd-standalone-tmpfiles is a sub-package of systemd. It is defined as a preferred requirement within samba. On the other side it is purposefully meant to conflict with systemd of same version or higher.

Let's start with systemd-standalone-tmpfiles. It is an optional requirement for samba-common(pulled in by samba-client) defined as below in samba.spec:
Requires(post): (systemd-standalone-tmpfiles or systemd)
and systemd is a requirement for base samba package(pulled in ultimately by samba-test)
Requires(post): systemd
Requires(preun): systemd
Requires(postun): systemd
At the same time systemd.spec has the following conflict defined in a weird way:
Conflicts: %{name}-standalone-tmpfiles < %{version}-%{release}^
This means that systemd-standalone-tmpfiles can only be marked for install if and only if its version is higher than installed systemd. fedora:36 base image comes with a particular version of systemd-libs installed and till 2 weeks back(Samba Container Image CI was running fine then) there were no updated packages. Around that time our samba-client container build process used to install systemd instead of systemd-standalone-tmpfiles and subsequent samba-toolbox build could succeed with already installed systemd.

With newer/updated packages(for systemd) we started installing systemd-standalone-tmpfiles instead of systemd while building samba-client container. Following that samba-toolbox container build process tries to install systemd resulting in a conflict because same binary(/usr/bin/systemd-tmpfiles) is already installed via systemd-standalone-tmpfiles.

Due to above mentioned reasons installing both samba-test and samba-client packages directly within fedora:36 base image will resolve all dependencies by pulling systemd(and not systemd-standalone-tmpfiles) and related packages.

@anoopcs9 Many thanks for this details and non-trivial info. I was wondering if ceantos:stream9 is less fragile in that respect.