raspbernetes/multi-arch-images

Add support for ceph

xunholy opened this issue · 8 comments

xunholy commented

Rook-Ceph itself supports ARM64 https://hub.docker.com/r/rook/ceph/tags however, the helm chart has other dependency images which are not all ARM64 supported.

quay.io/cephcsi/cephcsi:v2.1.0 ✅
quay.io/k8scsi/csi-node-driver-registrar:v1.2.0 ❌
quay.io/k8scsi/csi-resizer:v0.4.0 ❌
quay.io/k8scsi/csi-provisioner:v1.4.0 ❌
quay.io/k8scsi/csi-snapshotter:v1.2.2 ❌
quay.io/k8scsi/csi-attacher:v2.1.0 ❌

billimek commented

I wonder if @jamesorlakin could share whatever build script or automation they used to create the excellent multiarch csi images?

Perhaps we could leverage that approach for this repo?

xunholy commented

It would be awesome if @jamesorlakin was happy to raise a PR and consolidate his work into this repository for the community!

jamesorlakin commented

Hi all, thanks for the tag - I didn't know this project existed and will be handy for a Pi K8S cluster!
For the images I made in rook/rook#4051 (comment) these were very simple (and, well, unfortuantely manual) builds produced using docker buildx. These are all Go projects so I imagine cross compilation is fairly painless if there's an alternative in mind.

The setup for it all wasn't very obvious I thought, so here's the rough steps I performed on my spare Linux laptop:

  • Enabled Docker's experimental mode
  • Cloned each relevant Ceph CSI repo and checked out the latest released version
  • Created/tweaked a multiarch Dockerfile for each repo
    • I won't take responsibility for this one. There was a multiarch Dockerfile added in kubernetes-csi/node-driver-registrar#68 and I ended up reusing it everywhere with a different executable name in the copy step.
  • Setup quick and dirty buildx from a StackOverflow snippet:
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker buildx rm builder
docker buildx create --name builder --driver docker-container --use
docker buildx inspect --bootstrap
  • Logged into Docker Hub
  • Run the build on each Git repo, e.g.: docker buildx build -f Dockerfile.multiarch -t "jamesorlakin/multiarch-csi-snapshotter:2.1.1" --push --platform linux/arm64,linux/amd64,linux/arm/v7 .

You'll likely see from the above I'm very much new to the world of multiarch images. The Ceph CSI itself has ARM64 support but under a different tag. I believe this is due to a limitation of Quay.io but it makes a multiarch cluster powered by an operator slighly difficult. For that image I pulled both, retagged them locally, and then did something with docker manifest that I can't quite remember!

I built these images for armv7 too but Rook, Ceph, and Ceph CSI don't seem to have any support. I would guess that isn't crucial yet.

I'm more than happy to share the Dockerfiles and have a poke around with this repo later in the week. 🙂

billimek commented

I know this was closed, but we still don't have a multi-arch solve for the quay.io/cephcsi/cephcsi image. It is still single-arch (they to build a separate arm64 image, but it's not multi-arch).

edit I see that the ceph CSI image may be handled in a different PR: #89 (comment)

jamesorlakin commented

Oops, I agree on reopening for now. I probably shouldn't have put the "closes" keyword in the PR - sorry about that! Unfortunately I won't have any real time until later in the week or even weekend.

xunholy commented

Reopened issue @jamesorlakin @billimek.

Whenever you get time James that sounds awesome!

jamesorlakin commented

Edit: Magically it's all working now 🎉

It seems the Docker CLI used in CI doesn't have experimental mode switched on to support manifest operations. I'd incorrectly assumed the buildx setup would enable that.