Ensure support for building base images via Dockerfile

Question

Ensure support for building base images via Dockerfile

Opened this issue 8 months ago · 11 comments

It'd be nice if something like this worked:

FROM quay.io/centos-boot/builder as builder
RUN --mount=type=bind,rw=true,src=.,dst=/buildcontext,bind-propagation=shared rpm-ostree compose image --format=ociarchive --initialize manifest.yaml test.ociarchive
FROM oci-archive:/test.ociarchive

There's hints it is supposed to per https://github.com/openshift/os/blob/master/docs/rhcos-oci-braindump.md but I'm not seeing anything show up in the build context, the build fails very quickly on a nonexistent dir.

Now obviously we can break this into multiple containers per that doc, but that means the build is no longer just podman build.

Answer 1 · 2023-11-06T22:57:34.000Z

Note that rpm-ostree constructs the full OCI structure including layers on its own, that's the reason we want it to stay in control here.

Answer 2 · 2023-11-06T23:26:49.000Z

Can you be more specific about which part breaks or is a hypothetical? Is it a privilege limitation, or that the second stage is attempting to use, as its base, an archive file from an earlier stage, or something else I'm not spotting?

Answer 3 · 2023-11-07T14:38:28.000Z

or that the second stage is attempting to use, as its base, an archive file from an earlier stage

This.

Answer 4 · 2023-11-07T17:16:38.000Z

I don't think that's something the Dockerfile syntax really provides a way to express, but we treat the FROM argument loosely enough that, using RUN --mount to force the first stage to be built before the second when when it would otherwise be skipped or not necessarily be built in the desired order, you could get most of the way there:

ARG BUILDCONTEXT=.
FROM quay.io/centos-boot/builder as builder
RUN --mount=type=bind,rw=true,src=.,dst=/buildcontext,bind-propagation=shared rpm-ostree compose image --format=ociarchive --initialize manifest.yaml /buildcontext/test.ociarchive
FROM oci-archive:${BUILDCONTEXT}/test.ociarchive
RUN --mount=type=bind,from=builder,src=.,target=/var/tmp :

You'd have to build with --build-arg BUILDCONTEXT=... if you didn't start the build from the context directory, and I expect docker build would reject it, but it appears to succeed with the podman and buildah we have now.

Answer 5 · 2023-11-07T21:59:57.000Z

This is super interesting and has ramifications beyond wrapping rpm-ostree base image building. It means we could provide alternative ways to build container images while still being compatible with the builders out there that support this by using a Dockerfile stub behind-the-scenes. So then we could have the concept of "builder images" similar in some ways to Buildpack (or OpenShift's S2I).

Would it make sense to formalize it a bit so you don't have to play tricks to force ordering or require a --build-arg? I can file a buildah RFE if so.

For direct CLI usage, we could have a UX like

podman build --builder quay.io/centos-boot/builder /my/context/dir

where it would pass the place to output the OCI archive to the builder image via a build arg.

Answer 6 · 2023-11-08T14:59:05.000Z

I think I need to stress the "off the beaten path" nature of this. Adding affordances for it would imply a certain level of encouragement, and I'm not sure we want to sign up for that when the standard advice for jobs that can't be done with a Dockerfile is to use buildah's other commands.

Answer 7 · 2023-11-08T15:55:49.000Z

@nalind Yes. However one problem this solves is basically that saying "use buildah" opens up the big question of "which buildah from where" - Dockerfile is already a language to define that.

Obviously one can encode tooling using buildah in things like Github Actions, Tekton pipelines etc. etc. But I can't podman build a Github action YAML natively. This is basically about defining ways to build base images in that way.

Answer 8 · 2024-03-22T21:12:56.000Z

So @nalind I was playing around with this again, trying to minimize your example to e.g.:

ARG BUILDCONTEXT=.
FROM quay.io/centos-bootc/bootc-image-builder:latest as builder
COPY . /src
WORKDIR /src
RUN --mount=type=bind,rw=true,src=.,dst=/buildcontext,bind-propagation=shared skopeo copy docker://quay.io/fedora/fedora:39 oci-archive:/buildcontext/dest.ociarchive
FROM oci-archive:${BUILDCONTEXT}/dest.ociarchive
RUN --mount=type=bind,from=builder,src=.,target=/var/tmp :

Trying to build that I hit:

time="2024-03-22T20:52:07Z" level=fatal msg="committing the finished image: creating tar file \"/buildcontext/dest.ociarchive\": open /buildcontext/dest.ociarchive: permission denied"
Error: building at STEP "RUN --mount=type=bind,rw=true,src=.,dst=/buildcontext,bind-propagation=shared skopeo copy docker://quay.io/fedora/fedora:39 oci-archive:/buildcontext/dest.ociarchive": while running runtime: exit status 1

And yeah, it works with podman build --security-opt label=disable. With that I get:

 podman build -t localhost/test-container -f Containerfile .                                                                                                                                   125 03/22/24 16:51:55 PM
[1/2] STEP 1/4: FROM quay.io/centos-bootc/bootc-image-builder:latest AS builder
[1/2] STEP 2/4: COPY . /src
--> Using cache 9e814a42078b662af51fcaa3a23fc86d3f5a1dae346331e18ebdc9eb88bd747a
--> 9e814a42078b
[1/2] STEP 3/4: WORKDIR /src
--> Using cache 9ac9b60aca5dbc01fb7956226a821dd7b4372071e119097ba19201aef846a584
--> 9ac9b60aca5d
[1/2] STEP 4/4: RUN --mount=type=bind,rw=true,src=.,dst=/buildcontext,bind-propagation=shared skopeo copy docker://quay.io/fedora/fedora:39 oci-archive:/buildcontext/dest.ociarchive
--> Using cache 6342d0ee63592dfbd26e840b10c9d5da5d4d45faffad033fac0f2effb49fbc17
--> 6342d0ee6359
[2/2] STEP 1/2: FROM oci-archive:./dest.ociarchive
Error: creating build container: creating temp directory: archive file not found: "/dest.ociarchive"

So there must be some magic missing.

Answer 9 · 2024-03-25T16:15:50.000Z

So there must be some magic missing.

A z or Z option for the --mount flag. I have no idea why it's documented for the equivalent CLI flag for buildah run but not for the file.

Answer 10 · 2024-03-25T23:56:05.000Z

Hmm, actually the problem seems to be that this flow doesn't work with podman-remote (which isn't surprising I guess given the subtleties). I was trying those original commands on MacOS.

OK so...as you say, it seems like this support is not really documented really other than implicitly/indirectly via this statement in the buildah changelog:

The buildah from command now supports pulling images using the following three transports: docker-archive, oci-archive, and dir, as well as normal container registries and the docker daemon.

I'm not finding many tests for it either. So...I guess we're looping back to

Adding affordances for it would imply a certain level of encouragement, and I'm not sure we want to sign up for that when the standard advice for jobs that can't be done with a Dockerfile is to use buildah's other commands.

But the problem is there's just so much code that expects, processes, and operates on Containerfile. Would you be OK with patches adding some docs and tests for this and hence committing to supporting it?

Answer 11 · 2024-03-26T13:32:29.000Z

But the problem is there's just so much code that expects, processes, and operates on Containerfile. Would you be OK with patches adding some docs and tests for this and hence committing to supporting it?

That'd ensure that breaking those use cases would be intentional rather than accidental, which I'm on board with. Keep in mind that I don't expect other tools that accept Dockerfiles to support this sort of thing -- we get it for relatively little work by having containers/image do most of the heavy lifting.