Where did the built multi-platform image go?
Rory-Z opened this issue · 37 comments
I really like buildx, I want to use it in my code, but I have a issue.
I executed the docker buildx build --platform=linux/amd64,linux/386 -f . --output=type=image command and output the following information.
=> exporting to image 0.1s
=> => exporting layers 0.0s
=> => exporting manifest sha256:f70f46db5ab5126060072198f9fe4056240cf5ab9f0819a60bc141501d5b1198 0.0s
=> => exporting config sha256:af6012ceb069e31c852bb3e509eaab5cfdc1fca82d336e1a014cc7672989bcf6 0.0s
=> => exporting manifest sha256:b71684386f12acf835cf00fe1c1de7be104535d6debca08cf7789430c1d53456 0.0s
=> => exporting config sha256:0847da7ba34f27312e9e16eb015a309b290322c5d99ee64d56a172a75c906423 0.0s
=> => exporting manifest list sha256:4eb73fed7ba678c004b851cefcf2c8d9e5b60ce8bfceb3df09f32c08fbdd0296 0.0s
But I can't find my image. Where did it go?
Add push=true to the output or use --output type=registry to push the image to registry during build to access it.
@tonistiigi For some reason I don't want to push the image to the registry. Is there any other way?
There are other outputs as well. https://github.com/docker/buildx#-o---outputpath-typetypekeyvalue I pointed you to registry because you are building a multi-platform image, therefore I assumed you need to distribute it to multiple machines.
@tonistiigi I tried to use --output=tar, but it outputs a directory structure instead of a tarball. What should I do?
$ tree tar -L 2
tar
├── linux_386
│ ├── bin
│ ├── dev
│ ├── etc
│ ├── home
│ ├── lib
│ ├── media
│ ├── mnt
│ ├── opt
│ ├── proc
│ ├── root
│ ├── run
│ ├── sbin
│ ├── srv
│ ├── sys
│ ├── tmp
│ ├── usr
│ └── var
└── linux_amd64
├── bin
├── dev
├── etc
├── home
├── lib
├── media
├── mnt
├── opt
├── proc
├── root
├── run
├── sbin
├── srv
├── sys
├── tmp
├── usr
└── var
What way do you wish to access the image?
I want to be able to access multi-platform built-in images directly in the docker images. If not, I want to be able to export the image to a file, similar to the work of docker save. I guess --output = tar is equivalent to docker build && docker save, is that the case?
Docker does not support multi-platform images locally atm. Local image extracted in docker can only be for a single platform that the current node is based on. --output type=oci gives you the oci transport tarball with layers for all subimages.
I guess --output = tar is equivalent to docker build && docker save, is that the case?
No, that would be --output type=docker (with the limitations listed above)
I use --output=oci , but I don't see how it differs from --output=tar
$ tree oci/ -L 2
oci/
├── linux_386
│ ├── bin
│ ├── dev
│ ├── etc
│ ├── home
│ ├── lib
│ ├── media
│ ├── mnt
│ ├── opt
│ ├── proc
│ ├── root
│ ├── run
│ ├── sbin
│ ├── srv
│ ├── sys
│ ├── tmp
│ ├── usr
│ └── var
└── linux_amd64
├── bin
├── dev
├── etc
├── home
├── lib
├── media
├── mnt
├── opt
├── proc
├── root
├── run
├── sbin
├── srv
├── sys
├── tmp
├── usr
└── var
Post full commands of what you are running.
docker buildx build --platform=linux/amd64,linux/386 -t emqx/emqx:test -f deploy/docker/Dockerfile . --output=oci
--output type=oci https://github.com/docker/buildx#-o---outputpath-typetypekeyvalue
Hi @tonistiigi , I'm having the same issue, and may be trying to solve the same problem as @zhanghongtong .
Our goal is to export each of the separate built images locally to the docker daemon, and validate them locally (using https://github.com/multiarch/qemu-user-static and various test cases) before pushing them to the registry.
With --output type=oci I'm getting:
failed to solve: rpc error: code = Unknown desc = oci exporter cannot export named image
Is this possible at all without this kind of approach (given our Dockerfile is under a subdirectory named $VERSION):
for ARCH in amd64 arm64v8 ppc64le; do
from="$(awk '$1 == toupper("FROM") { print $2 }' $VERSION/Dockerfile)"
docker pull "$ARCH/$from"
docker tag "$ARCH/$from" "$from"
docker build -t apache/couchdb:$arch-$VERSION $VERSION
done
allowing access to each separate platform image as apache/couchdb:$arch-$version locally for validation, then assembling the manifest and pushing later?
Building all of the images at once, and bringing them over one at a time for testing would be acceptable, too.
What would be the approach pushing the results of the docker buildx build (images + manifets) to a local registry, eg. Nexus ?
The --push argument allows only pushing to the Docker Hub.
It tried --output=type=registry,ref=localhost:5000 but that is not recognized, still requiring authentication at the Docker Hub:
failed to solve: rpc error: code = Unknown desc = server message: insufficient_scope: authorization failed
Or is there some other solution using intermediate folders or archives ?
localhost:5000 is not a valid image ref. it is probably interpreted as a hub image.
You mean ref= exists for --output=type=registry ? This is not documented.
I was thinking of adding the registry destination URL, similar as if just giving the --push argument (shorthand for --output=type=registry, see documentation) which is using docker.io by default. localhost:5000 is the address of my local registry
The image ref. should be provided/added by the buildx build command.
@DannyBoyKN have you tried just tagging your image as if you were going to push to you r local repo? E.g.
docker buildx build -t localhost:5000/marchpkg:latest --platform linux/amd64,linux/ppc64le . --push
This is how docker knows the host. Admittedly, I tried to do this for a multi-arch image I'm trying to build, but I'm not getting far enough to push, yet. Good luck.
Honestly, I don't remember if I tried this, I think I did ...
Unfortunately, I'm stuck, too!
The build process already fails when downloading because of the DNS. buildx uses by default google (8.8.8.8 and 8.8.4.4) which is not available behind my firewall. Giving the local DNS in the daemon.json file stops with 'connection refused'
I'll try further ...
Well, just retried - was sure I did it already - with this Dockerfile:
FROM gcc:4.9
COPY main.c /usr/src/myapp/
WORKDIR /usr/src/myapp
RUN uname -m
and then
>$ docker buildx create --use --driver docker-container --name multiarch
...
>$ docker buildx build --platform linux/amd64,linux/arm64/v8,linux/arm/v7 --tag localhost:5000/multiarch:test --push .
the error is
...
------
> exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed to do request:
Head http://localhost:5000/v2/multiarch/blobs/sha256:39f6cc0761da5c1bc61d59c5cbe9188f22bc173d6f1038d6cccf1292f0b79594:
dial tcp localhost:5000: connect: connection refused
If you are pushing to localhost from a container driver you need to use host networking for the container https://github.com/docker/buildx#--driver-opt-options . Custom dns can be set with buildkitd config file.
i need also to test my image and push them later
for arch in amd64 arm64 arm ; do
docker buildx build \
--platform $arch \
--output type=docker \
--tag me/myimage:${version}-${arch} \
$version/
done
It works well
Using --driver-opt network=host indeed works for pushing to localhost. But that's only for my local testing. The aim is to get them pushed to our Nexus registry, but still don't get the DNS configured.
With the "buildkitd config file" you mean in ~/.docker/config.json ?
Whatever DNS is set there it is correctly propagated into the conatiners /etc/resolv.conf. I did several DNS settings with and without network=host and didn't succeed ... don't have the exact error at hand at the moment ....
That's interesting and tagging, pushing and running with localhost:5000/gcc-4.9:${arch} --push worked:
FROM gcc:4.9
RUN uname -m$ docker run --rm localhost:5000/gcc-4.9:arm uname -a
Unable to find image 'localhost:5000/gcc-4.9:arm' locally
arm: Pulling from gcc-4.9
e925dd4ffa2a: Pull complete
c9bfbf7dfc78: Pull complete
015138dd660d: Pull complete
d88b2b5023e5: Pull complete
4d0d77a38079: Pull complete
996bfab2b29c: Pull complete
d27243b445c7: Pull complete
2f949e025be6: Pull complete
d55a5da9fec4: Pull complete
3976cacabfa7: Pull complete
Digest: sha256:b8dcfe0a3bbf2dbcb49a5117d8dee8fd412da31663a8c9be745eb6909bebf4d2
Status: Downloaded newer image for localhost:5000/gcc-4.9:arm
Linux c07606921642 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 armv7l GNU/Linux
@DannyBoyKN were you able to figure this out? I have the same problem which is a multiplatform build using buildx but I am not able to specify the --push flag because I am trying to push to a private nexus registry
This is my command :
docker -D buildx build --platform linux/arm64,linux/amd64 -t private.repo.com/nav_2_0:multi_support_image --push .
Unfortunately not. I had not time so far to dig into howto provide the correct DNS information as @tonistiigi pointed out above.
Why is this issue closed? How can you do a multi-arch build with buildx now, and save the image, without pushing it to any registry?
@ballerburg9005 i use both :
1 - use buildx with --output "type=docker,dest=myimage.tar"
2 - use buildx with --output "type=oci,dest=myimage.tar"
for arch in amd64 arm64 arm ; do
docker buildx build \
--platform $arch \
--output "type=docker,push=false,name=me/myimage:mytag-$arch,dest=myimage.tar" \
$path_to_dockerfile/
done
Documentation can help as well :)
Not sure why docker buildx build doesn't save a copy to local image as default. After run it, I don't see it locally.
So what's option to save the image as local image, then I can docker image |grep <image> directly?
@ozbillwang --output "type=docker,name=${app_name}:${tag}
can't make it work, any helps are appreciated.
$ docker buildx build --platform linux/arm/v7,linux/arm64/v8,linux/amd64 --output "type=docker,name=demo" .
> exporting to oci image format:
------
error: failed to solve: rpc error: code = Unknown desc = docker exporter does not currently support exporting manifest lists
If i switch to --load, shorthand for --output=type=docker, got same error.
Updates
finally, got the issus.
if build locally, no need provide --platform.
$ docker buildx build --load -t demo .
$ docker images |grep demo
demo latest bad643520ce9 12 minutes ago 48.2MB
If --platform has multiple values, buildx build will also create a multi-arch manifest, which is not compatible with --load/--output=type=docker. The solution from @barcus worked because the for loop did each architecture one at a time. If you want to use buildx build and have the image available in the local registry, either don't specify a platform, or only specify one at a time.
Is there any official documentation on how to handle this use case? I think its fairly common to want to build a multi-arch image in CI, and not being able to test it in a straightforward manner really complicates what I think is a standard procedure.
not really official but using github action at least :)
https://dev.to/cloudx/multi-arch-docker-images-the-easy-way-with-github-actions-4k54 ?
Okay, so posting here because question is attached.
The build took a LOT of time, now I want to push it afterwards without redoing it. It's also a multi-arch build, so yes, it tells me that two manifests and a manifests list is created - also with hash values. But I see NO WAY to use these hash values...
The relevant part was -o type=image,name=qt6.3-gcc but git push qt6.3-gcc tells me that
Using default tag: latest
The push refers to repository [docker.io/library/qt6.3-gcc]
An image does not exist locally with the tag: qt6.3-gcc
So how can I refer properly to push already created image?
Edit:
Ok, apparently the build commit creates a container instead of an image, so a commit is needed before, then push works. Let's see if this is the idea...
Edit:
But it only uploaded one architecture... how can I upload all architectures?
Edit:
I tried using the manifest hash values but they are all not recognised. It will give me even errors like "authentication needed", which makes no sense, I think, I'm currently just hoping to create a manifest list locally... also, where on macOS filesystem are the manifests stored? No chance to find them currently. I only see Docker.raw file which is huge but useless in that shape.
So what actually IS it doing if its not being pushed, and not being stored.
Wouldnt it be better to error out if the result is to be discared upon completion? Or at least inform the user "Hey, I threw the build artefact away, google the correct way to do this"?
So, my 5 cents here...
Sometimes it makes sense to push later for various reasons:
- you could be without internet.
- it could be that you want to create it as a basis and build/add other images later to the manifest before pushing
- a third use case is that you build first and care later where to push, maybe you are just setting up your environment and are not ready. It could still make sense to start a long-running build task at this point
Just in general, for any build/IT tool out there, building should be a distinct step from deploying. It's nice if it can be combined, but that is optional. It should not be that splitting it is the optional one. Maybe, I have some different background, but to me it seems that this is the general expectation to such tools, so breaking with it seems inconsistent, thus anyways violating "the principle of least surprise" (I might on that one stretching it a bit, but yeah, it surprised me)
Gladly take with a grain of salt, all just my opinion :)
Edit: That is, if storing it locally is hard to achieve than an error/warning message that the result will be unusable is definitely better than nothing! So maybe go with this first if it's still unclear on your side whether you will offer local storage?
You can try the multi-platform load with https://docs.docker.com/desktop/containerd/
What way do you wish to access the image?
I'm a bit surprised by this question. The older images didn't need to deal with multi-architecture. Now that we've moved into that direction, we want to confirm that images we've built are correctly supporting multiple architectures before pushing it up. (And I'm sure there's a lot of other reasons)