Some images tagged with `linux/arm64` arch actually contain `linux/amd64` builds
strophy opened this issue · 4 comments
Running some sbtscala/scala-sbt images based on upstream Java8 or Java11 fails under aarch64 as follows:
ubuntu@ip-172-31-18-238:~$ uname -m
aarch64
ubuntu@ip-172-31-18-238:~$ docker manifest inspect sbtscala/scala-sbt:graalvm-ce-21.2.0-java8_1.8.0_2.12.17
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 2835,
"digest": "sha256:5830ea12b7fab65e1b59ed2de65ebc49cc178cbdf2a2c9df8d5bc28c745281ae",
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 2835,
"digest": "sha256:bb5fede2ab7269231d4bd43464054b2be17ef9249778cd6a207d0c50c08b7e25",
"platform": {
"architecture": "arm64",
"os": "linux"
}
}
]
}
ubuntu@ip-172-31-18-238:~$ docker run -it sbtscala/scala-sbt:graalvm-ce-21.2.0-java8_1.8.0_2.12.17 /bin/bash
exec /bin/bash: exec format error
The same command succeeds under x86_64:
ubuntu@ip-172-31-47-214:~$ uname -m
x86_64
ubuntu@ip-172-31-47-214:~$ docker run -it sbtscala/scala-sbt:graalvm-ce-21.2.0-java8_1.8.0_2.12.17 /bin/bash
bash-4.4#
This is occurring because CI in this repo specifies the baseImageTag as the upstream ghcr.io/graalvm/graalvm-ce container repository. However, this upstream repo does not provide linux/arm64 builds for the Java8 and Java11 base images used in the Dockerfile here. Compare the output for the following tags, only Java17 returns an actual multiarch manifest:
java8-21.2.0(upstream single arch manifest)java11-21.3.0(upstream single arch manifest)java17-21.3.0(upstream multiarch)
docker-protobuf on feat/scala [!] ❯ docker manifest inspect ghcr.io/graalvm/graalvm-ce:java8-21.2.0
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:b6b7306e93599a008905837de57727f173089c1bcdf921b205a176923e336365",
"size": 4835
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:1da50e1664e1b58d93182fe5af093c642668ec93f67fcd8215b9c1979930b84d",
"size": 42178827
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:60181773193d5f43c673f020d3e462523267ff1ced388a64d53e264aa02d2894",
"size": 113567674
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:1ed478fbf2d5dcb061baa77a06c80c20afe132e65a5ce58015ca8548fdc89820",
"size": 1406
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:d0be10971ce1d959ad3bd39a2cc5e0b9e1e2e9f4c2c62d1bd3bc5e7a044df4f7",
"size": 562
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:2d574c1e25083654dba1ed6b2ebb2b509e92be2960a33ce54d680e14845e80ea",
"size": 299154725
}
]
}
docker-protobuf on feat/scala [!] took 2s ❯ docker manifest inspect ghcr.io/graalvm/graalvm-ce:java11-21.3.0
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:9c09d46390b65b4cf89c887f3d0ed901d6f2ace9ff4b48b92696fe9ba9a0d71e",
"size": 4840
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:58c4eaffce77ac1fb013bf82c91927c631802ad54465ebc9b687b5dc8ee73c02",
"size": 41961111
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:ffb226b40205bd44e5974716ee2d01cad523fab8ba34fba366e9d4b2436c177b",
"size": 114316989
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:05f16e1f6df1141244dc840c478a490ebfc0760ed7707b9a25bcb1f60a8a6b0a",
"size": 1412
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:69b5e9a658d59404f6878f0cceee5ef4ee8c69ebad90fc5f150fe1e4075f4e77",
"size": 557
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:1614f6f1cf15325afb47f4b43c21ac1bba368ce8b15db5970e89d112b1062976",
"size": 410399609
}
]
}
docker-protobuf on feat/scala [!] took 2s ❯ docker manifest inspect ghcr.io/graalvm/graalvm-ce:java17-21.3.0
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 1370,
"digest": "sha256:d3877ce3bae9538032f02d88fb5ea5019d5686a15dd15615fb4e9c65970c72cc",
"platform": {
"architecture": "arm64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 1370,
"digest": "sha256:0e93350c4c5e1797fd02a34c2dd373e8783fc500b08a06f4082c57503779e3d4",
"platform": {
"architecture": "amd64",
"os": "linux"
}
}
]
}
The result is that the linux/arm64 tag is being coerced onto a container built for linux/amd64 arch. This can also be seen in the identical size of the images being published, for example (note the size of both images):
This problem does not appear in the test action step, because only the linux/amd64 image is tested.
I'm not sure if the solution here is to ask upstream to publish linux/arm64 images for you to depend on, or to remove support from your build system. I'm also not sure if other base images are affected, I only looked at GraalVM with 3 different Java versions. Took me a while to track this down, so I thought I'd raise an issue to ask 😸
Thanks for your detailed report, looking into this
I guess the main issue is docker/buildx#844 where buildx should fail instead of using the wrong image
Thanks for the quick response! It looks like they are discussing breaking backwards compatibility in Docker and making this sort of build finally throw an error. Worth it in this case, imo - Docker shouldn't be happily pushing images for the wrong arch like this.
I'll try to find another solution for building Scala on ARM after discussing with devs.