Add experimental support for unison via native Apple M1 compatible image
EugenMayer opened this issue · 23 comments
To speed up unison a bit, add support for using a native ARM64 build for Apple M1.
This is possible sinc 0.6.0 by manually adjusting the yet experimental image
To use the ARM64 build https://docker-sync.readthedocs.io/en/latest/getting-started/configuration.html#sync-strategy-image to eugenmayer/unison:2.51.3-4.12.0-ARM64
When it got more battle proven, consider how we could default to ARM64 images when we detect that the arch type is M1.
This is where i would need contributions for, how would we detect the arch in docker-sync so we can change the unison image dynamically
I tried to change the image to this new one, but still get the warning that it's not arm64. If you check on dockerhub it says linux/amd64, too (https://hub.docker.com/r/eugenmayer/unison/tags?page=1&ordering=last_updated&name=ARM).
Unable to find image 'eugenmayer/unison:2.51.3-4.12.0-ARM64' locally
2.51.3-4.12.0-ARM64: Pulling from eugenmayer/unison
339de151aab4: Pull complete
d9d90b7eacac: Pull complete
263c8fcec158: Pull complete
4662243b0ec1: Pull complete
5cb20a37c167: Pull complete
58c5100d8be9: Pull complete
efe86f261a6a: Pull complete
3da451fd04e1: Pull complete
bfa3b0268263: Pull complete
Digest: sha256:b784479dcd3d26d3dda8e164e79d749a04808cd38c02d6b06e1cac64624a3945
Status: Downloaded newer image for eugenmayer/unison:2.51.3-4.12.0-ARM64
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
doing initial sync with unison
This is where i would need contributions for, how would we detect the arch in docker-sync so we can change the unison image dynamically
Shouldn't docker pull the correct arch by itself if it's available? At least that's how it works for all the other images. It checks if arm64 is available, if not it fallbacks to amd64 and emulates it on M1.
I guess i will need to build the image under ARM for ARM in travis.
Could you try building the image on your M1 once to confirm
git clone https://github.com/EugenMayer/docker-image-unison
cd docker-image-unison
docker build ---build-arg "BASE_IMAGE=arm64v8/alpine:3.12" -build-arg "OCAML_VERSION=4.12.0" --build-arg "UNISON_VERSION=2.51.3" -t custom-docker-image-unison -t ziro-unison
then switch the unison to ziro-unison and try again?
Thanks
@EugenMayer BASE_IMAGE has to be set to amd64? Or do you want to see if it's able to built an amd64 version?
sorry, arm64 - corrected it
ok now it all makes more sense - it should have been 'arm64v8' all the way down. corrected it for you
"BASE_IMAGE=arm64v8/alpine:3.12" works. It's the same like building the master without build-args on M1 (the image i'm currently using). no warnings, everything works fine.
does docker-sync work with that unison? Current effort at EugenMayer/docker-image-unison#23
I'm using the image for 4 days now and didn't notice any problems (build from master 4 days ago, without any changes, plain docker build . -t docker_unison). I'm using native_osx sync strategy, because unison fails for me.
unison is used in native_osx, just in the container (not from osx to container, but inside).
You might want to try the newly build images which might speed things up since they are native arm:
docker pull eugenmayer/unison:2.51.4_rc2-4.12.0-ARM64
Anyway thanks for reporting, that issue should be fixed now
.travis.yml
script: docker build --build-arg "OCAML_VERSION=$OCAML_VERSION" --build-arg "UNISON_VERSION=$UNISON_VERSION"
. -t eugenmayer/unison:$UNISON_VERSION-$OCAML_VERSION-$ARCH
the BASE_IMAGE build-arg is missing, isn't it.
you are right, it was missing. No idea how i tested all that. Added it, thanks for the hint
Using the <sync_strategy>_image config to explicitly use the ARM64 image works for both the native_osx and unison sync strategies on my M1. My issue now is supporting the same project for Intel CPU macs and Apple silicon macs requires different YML files to overwrite the sync_strategy_image specifically on the M1 macs.
@EugenMayer Would you consider adding docker manifests to the docker-image travis builds? For example, a manifest would be created with the eugenmayer/unison:$UNISON_VERSION-$OCAML_VERSION tag and annotated with the eugenmayer/unison:$UNISON_VERSION-$OCAML_VERSION-AMD64 and eugenmayer/unison:$UNISON_VERSION-$OCAML_VERSION-ARM64 tags/images. Then I could set my sync_strategy_image to the manifest tag (or we could change the docker-sync default), and at runtime my docker daemon will pull the correct architecture image regardless of if I'm on intel or apple silicon.
https://docs.docker.com/engine/reference/commandline/manifest/
TLDR;
#docker-sync.yml
version: "2"
syncs:
your-files-sync:
sync_strategy: 'unison'
unison_image: eugenmayer/unison:2.51.3-4.12.0-ARM64
notify_terminal: true
src: '.'eugenmayer/unison:2.51.3-4.12.0-ARM64 is found here https://hub.docker.com/r/eugenmayer/unison/tags?page=1&ordering=last_updated&name=ARM to get latest tags.
@DominicChiampi not planning to support those manifest.
You are in need of overrides here, since you use different arches for macOS - we do not have something like that implemented yet AFAIU. So either you use different docker-sync.yml files or we find a new way to check that if
if macos
if apple M1
use ARM
otherwise
use amd64
end
This needs to be patched in the strategy itself IMHO
Resolving since we have Apple ARM M1 support and it seems to work for the people
I am currently worting on making our Dev Environment M1 ready and i noticed that in my case the default image that is pulled is a AMD64 not ARM. We use "native_osx" Strategy without any configuration regarding the image.
I suspect every Team with more than a few Persons is not able to upgrade their Hardware to have a consistent setup. So an automatic detection is essential to prevent manual modifications beeing necessary for every project that a M1 is working on...
@bweinzierl a PR for that detection is welcome anytime, happy to review it
@EugenMayer Never done any coding in Ruby, i don't feel comfortable to take a shot at it right now.
Just wanted to clarify that it still makes sense to detect automatically. Maybe we could sponsor some development on it... need to finish my analysis of necessary changes to our dev environment to get an overview that has to be done also to other software...
@DominicChiampi I think i have the same use case as you. Did you find another way?
@EugenMayer There may be a better solution then to have some kind of switch implemented in code. Is there a reason why the unison image is not provided as multiarch image? As i understand it the alpine base image is already multiarch. So the only thing that would have to be done is to build the image with buildx.
Did you try that already? Is there some obsticle i am not aware of?
I got some experience in the last few month because i had to provide multiarch images for our development environment. Maybe i can actually help there.
We are now using out own multiarch build of the unison image. Works great. Just needs to be built with buildx. No modification of dockerfile neccessary.
Later i found this:
https://hub.docker.com/r/moorecp/docker-image-unison/tags
Seems like somebody has already done it. If this for some reason does not work i can provide our version on docker hub. Just send me a message...
@EugenMayer any chance you could start building unison images with buildx?
I am getting this message after specifying the ARM image in the unison_image config item
Digest: sha256:aa11fb87406a5ef01883cc346852d0be7fa83525fbf4737321aae58432c3275c Status: Downloaded newer image for eugenmayer/unison:2.51.3-4.12.0-AMD64 WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Would it be possible to get a build of the image with linux/arm64/v8 ?
@EugenMayer have you considered multi-platform builds? I just got his working for some images that we are using locally on M1 and x86. It is easy to set up.
@jkampfwealthbox i'am currently not investing any time into maintaining docker-sync myself. Mainly since i do not own a Mac for development since 4 years and use linux - since this goes for our entire dev team, i do neither have the usual time at my hand to work on things, but more importantly, i do not even have a device to test any of the given things myself. Neither i have a MacOS device, nor i have any M1/M2 CPU based devices on top of that.
So to be honest, until that changes, i do neither have the motivation, but more, the hurdles are way to big to even try to go after those things.
So if you people want to move anything substantial, it goes through PRs only. The smaller chunks, the better - i try to make up time for reviewing those as much as possible.
Hope this gives a little transparency in what to expect to happen and what not!
Thank you all for already digging out so many details here.