containerd/nerdctl

What is required to support --platform

Closed this issue · 4 comments

jandubois commented

Not sure where to ask this; this seems like the closest place:

I was planning/hoping to use qemu & binfmt_misc for lima to support aarch64 containers on amd64 and vice versa because running the whole VM in emulation is very slow. This works fine with docker:

$ docker run --platform linux/amd64 busybox uname -a
[...]
Linux e76e01580924 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 x86_64 GNU/Linux

$ docker run --platform linux/arm64 busybox uname -a
[...]
Linux 748a60516af3 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 aarch64 GNU/Linux

But I haven't found a way to specify the desired platform with containerd/nerdctl/kubernetes. I couldn't even find any recent discussion of this topic. Is this something that sounds reasonably achievable, or is it missing basic support at the containerd level? If achievable, what would be the required steps?

jandubois commented

I see that ctr images pull at least supports --platform, which looks promising:

root@lima-default:/Users/jan# ctr images pull --platform linux/arm64 docker.io/library/busybox:latest
[...]
unpacking linux/arm64 sha256:930490f97e5b921535c153e0e7110d251134cc4b72bbb8133c6a5065cc68580d...

But somehow ctr run doesn't seem to work the way I expect it to (which is likely entirely my own fault).

jandubois commented

I've installed the whole qemu-user package:

root@lima-default:/Users/jan# ls /proc/sys/fs/binfmt_misc/
python3.9     qemu-arm    qemu-hppa        qemu-mips      qemu-mipsel     qemu-ppc      qemu-riscv32  qemu-sh4    qemu-sparc32plus  qemu-xtensaeb
qemu-aarch64  qemu-armeb  qemu-m68k        qemu-mips64    qemu-mipsn32    qemu-ppc64    qemu-riscv64  qemu-sh4eb  qemu-sparc64      register
qemu-alpha    qemu-cris   qemu-microblaze  qemu-mips64el  qemu-mipsn32el  qemu-ppc64le  qemu-s390x    qemu-sparc  qemu-xtensa       status

Normal amd64 binary runs as expected:

root@lima-default:/Users/jan# ctr run --rm --tty --platform linux/amd64 docker.io/library/busybox:latest busybox uname -a
Linux lima-default 5.11.0-18-generic #19-Ubuntu SMP Fri May 7 14:22:03 UTC 2021 x86_64 GNU/Linux

But arm64/aarch64 one doesn't:

root@lima-default:/Users/jan# ctr run --rm --tty --platform linux/arm64 docker.io/library/busybox:latest busybox uname -a
standard_init_linux.go:228: exec user process caused: no such file or directory

root@lima-default:/Users/jan# ctr run --rm --tty --platform linux/aarch64 docker.io/library/busybox:latest busybox uname -a
standard_init_linux.go:228: exec user process caused: no such file or directory

root@lima-default:/Users/jan# cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/libexec/qemu-binfmt/aarch64-binfmt-P
flags: POC
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

root@lima-default:/Users/jan# ls -l /usr/libexec/qemu-binfmt/aarch64-binfmt-P
lrwxrwxrwx 1 root root 22 Apr  7 09:58 /usr/libexec/qemu-binfmt/aarch64-binfmt-P -> ../../bin/qemu-aarch64

Maybe some library is missing?

AkihiroSuda commented

Please try sudo nerdctl run --privileged --rm tonistiigi/binfmt --install all

root@lima-default:~# nerdctl run --privileged --rm tonistiigi/binfmt --install all
docker.io/tonistiigi/binfmt:latest:                                               resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:ce4d5a2a6ac4a189047fca2d71cbd901cc7beebacf538be95fccb3aca87cb2ec:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:1299eebc7e528cf34e4422816eac358bdb945c152fee91d130c0e640c246f054: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:258d55b22af6c9b5d9e637ba4349db3902f6a51d00625228e553175fd2f2f0f1:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:6f3748a7914120920d1a455b3d87d9e3c33b8359520b2f98829aa2432403bd57:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:a5644ca467e15321e2343c15bf9149fb7354c8a0f682679c943387ccdfae74fe:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 4.5 s                                                                    total:  14.4 M (3.2 MiB/s)                                       
installing: arm64 OK
installing: arm OK
installing: mips64le OK
installing: mips64 OK
installing: s390x OK
installing: ppc64le OK
installing: riscv64 OK
{
  "supported": [
    "linux/amd64",
    "linux/arm64",
    "linux/riscv64",
    "linux/ppc64le",
    "linux/s390x",
    "linux/386",
    "linux/mips64le",
    "linux/mips64",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "qemu-aarch64",
    "qemu-arm",
    "qemu-mips64",
    "qemu-mips64el",
    "qemu-ppc64le",
    "qemu-riscv64",
    "qemu-s390x"
  ]
}
root@lima-default:~# uname -m
x86_64
root@lima-default:~# ctr run -t --rm --platform=arm64 docker.io/library/alpine:latest foo
/ # uname -m
aarch64
AkihiroSuda commented

Relevant ticket: #103