whalebrew/whalebrew

Add ARM/ARM64 support

farkasmate opened this issue · 14 comments

farkasmate commented

Goal: Run whalebrew on ARM/ARM64 architecture

Example: Run whalebrew on Raspberry Pi boards or WD NAS boxes

Benefits:

  • Provide alternative package management on embedded systems
  • Promote multiarch image development
farkasmate commented

I have a proof of concept on my WD EX2 Ultra NAS (armhf/arm32v7).
Depending on interest I can make it compatible with the mainstream CI pipeline.

Is the Travis pipeline the way forward? (Repo still has GitHub Actions.)

tjamet commented

Hi!
That sounds a great idea!
I am not super aware about how multi-arch works, I had in mind it is pretty transparent on the docker run side that whalebrew uses, am I wrong?

This though would mean a double build on whalebrew-packages to build for x84 and arm64 architectures.

Indeed, the repo is using GitHub actions, to build, test and package. Is this a blocker?

farkasmate commented

Hi @tjamet,

I'm not super experienced with multi-arch either, here's what I know:

  • Yes, transparent to docker run
    • If there's an arch match it's expected to work out of the box
    • If there's no matching arch the container is expected to fail with exec format error
  • whalebrew-packages could be built with buildx, shipped as experimental in the latest docker version
    • Technically double build, practically makes a single image

I start working on a fork and get back to you in a couple of days.

  • 👍1
tjamet commented

great! don't hesitate to ping me if you need
probably we can split it into:

  • building and publishing whalebrew for arm
  • start building packages multi-arch (having an optional arch in the package configuration?)
farkasmate commented

I agree, it makes sense to split it.

I managed to cross-compile whalebrew to Linux ARM and Linux ARM64 with these changes:
master...farkasmate:multiarch

A couple of findings I want to hear your opinion on:

  • Building for Apple Silicon doesn't seem to be possible at the moment, but should be possible with the next release of Golang. Should I leave it in place?
  • Build currently runs on ubuntu-latest meaning testing the MacOS binary by running whalebrew version is not possible anymore. Any idea how could we test it better?
  • The 4 built binaries will look like this
    • A bit messy in my opinion
    • Binaries originally had been named whalebrew-Darwin-x86_64 and whalebrew-Linux-x86_64. As uname outputs can't be used (cross-compiling) they are now in the whalebrew-darwin-amd64 and whalebrew-linux-amd64. We can map this back if necessary

On the topic of building multiarch whalebrew-packages I did some spots and found three different scenarios so far:

  • Just works like figlet, because the base image itself is multiarch with ARM and ARM64 versions available
  • Doesn't work with ARM/ARM64 like jq, because the Dockerfile is referencing the jq-linux64 binary
  • Doesn't work at all (?) like terraform
    • I couldn't build it to amd64 either
    • The referenced python-dev package is not available for hashicorp/terraform:latest which is based on Alpine Linux v3.12. The package was last available in Alpine Linux v3.4

Based on this I'm not sure how viable to build all/most whalebrew packages into multiarch images. Needs more investigation.
If you agree, I'd say to focus on

building and publishing whalebrew for arm

first.

tjamet commented

Building for Apple Silicon doesn't seem to be possible at the moment

We should then exclude it IMHO

running whalebrew version is not possible anymore. Any idea how could we test it better

Maybe a good incremental step is to just ignore it if: ${{ !startsWith(matrix.arch, 'arm') }}

Though, I think we should try to keep the compilation on the same OS

We can map this back if necessary

IMHO that's a good idea, it would allow keeping the same "contract" to publish binaries

I'm not sure how viable to build all/most whalebrew packages into multiarch images

Agree, we keep it small, just build a working version of whalebrew for arm, we will solve the package afterwards

stunthamster commented

FWIW I've built the binary using Go 1.16beta1, and it works fine as far as I can see - I'll be testing it for the rest of today, I'll let folks know if anything nasty pops out.

farkasmate commented

@stunthamster I noticed an error when tried to build the master branch with Go 1.15.
I'm working from 0.2.5 for this issue.

After weeks of silence, I'm going to rework my branch according to @tjamet's response.

  • 👍2
farkasmate commented

@tjamet In the #122 PR MacOS build is happening using the macos-latest image.
I excluded the whalebrew version test runs of ARM/ARM64 images as you suggested.

As far as manual testing goes the ARM image works on my Raspberry Pi 3B+ using Raspbian 9 and on my Western Digital EX2 Ultra NAS. The ARM64 image works on Raspberry Pi 3B+ using Ubuntu 18.04 LTS (64bit).
Testing was done with this multiarch figlet image I built using buildx (shipped with Docker 19.03).

If you want to give a test to any of the binaries yourself, they are available here:
https://github.com/farkasmate/whalebrew/releases/tag/0.2.5-test-tag-2

Ping me if you have questions.

Happy new year!

tjamet commented

Happy new year as well and thanks for the contribution @farkasmate I have just released it, you can find the binaries here: https://github.com/whalebrew/whalebrew/releases/tag/0.3.0

farkasmate commented

Awesome, thanks.

I started doing some experiments with multiarch images.
Do you want to keep discussion here or should I open an issue in whalebrew-packages?

tjamet commented

packages will indeed be harder, probably worth keeping the discussion here for now.

farkasmate commented

Hi @tjamet,

I wonder if there's still interest in extending the ARM/ARM64 support.

2 years passed since the last update here and a few things happened in this time.

New tickets related to issues we discussed above:
#135 - Terraform was broken then and still broken
#138 - Apple Silicon
#168 - Apple Silicon

I've used whalebrew on ARM a lot in the past 2 years and I also have an Apple Silicon Mac for a month now, which simplifies testing ARM64 a lot.

The following packages are broken if I try to rebuild them on x86:

  • azure-cli
  • bonnie++
  • mitmproxy
  • openshift-cli-v2
  • plantuml
  • review
  • seagull
  • terraform
  • travis
  • unrar

On ARM/ARM64 the following one's are also broken:

  • erd (broken on ARM as based on haskell, which only supports amd64 and arm64)
  • httpie (gcc error, might be fixable)

Because of lack of ARM support for erd I'm leaning towards your original idea of:

start building packages multi-arch (having an optional arch in the package configuration?)

What's more bothering is the broken packages on x86. They needs fixing before building them for ARM/ARM64 can be considered. I should be able to chip in, although I'm not familiar with many of the tools on the list.

One small caveat, if we use docker buildx for building multi-arch images, every image needs to be bumped to require whalebrew 0.3.0 as a minimum version due to: #121

Is this something worth working on again?

farkasmate commented

I'm closing it as completed.