Support for Windows on Arm

Question

Support for Windows on Arm

pbo-linaro opened this issue 3 years ago · 19 comments

Checklist

I added a descriptive title
I searched for other feature requests and couldn't find a duplicate (including also the type-feature tag)
I confirmed that it's not related to another project (see the above section)

What is the idea?

Hi, Linaro is currently working with Microsoft to enable a lot of open source projects to run on Windows on Arm.

We are currently investigating if conda could be easily enabled for this new architecture.
The main blocking point today is the lack of "free" Windows on Arm machines in CI systems.
Azure started deploying Arm server, but this is not a free solution.

Thus, we are targetting cross compilation for now, and we would like to experiment with it, before offering any official support.

Why is this needed

Microsoft is investing a lot of effort in this new architecture, and the numbers of users is growing steadily.

Since conda allows to distribute a lot of software, it is a great way to support a lot of projects through it.

What should happen?

So far, I'm trying to build a package for this new architecture on a custom channel at anaconda.org.

I added a conda_build_config.yaml:

target_platform:
  - win-arm64

I started with a recipe without any dependencies.
conda-build insists on finding ...\conda-bld\win-arm64\repodata.json, which, of course, is not available at the time.
I managed to bypass this by copying it (locally) from win-64 arch.

Is a support needed on anaconda side to make it work?

After that, I tried to compile a C package. Alas, due to the absence of repodata.json, no compiler package can be found.
Is there a way to define it locally, so I can at least try to build a few packages?

I didn't manage to find useful documentation on porting to new architecture, which, I can understand, is not very frequent.
Where is the best place to get information on this?

Thanks,
Pierrick

Additional Context

No response

Answer 1 · 2022-05-24T07:25:10.000Z

Polite ping :) Does anyone has experience with porting conda to new architectures?

Answer 2 · 2022-05-31T18:15:54.000Z

cc @chenghlee, @jezdez

Answer 3 · 2022-06-07T01:31:56.000Z

@pbo-linaro Enabling Windows on ARM support requires a few things, some easier than others.

Getting conda and conda-build to recognize Windows on ARM as an "official" platform/subdir. This is fairly easy, but we will need to know what CPython reports for sys.platform, platform.machine(), and platform.architecture() on this platform. (The biggest question I have at the moment is whether the CPU architecture is reported as aarch64 or arm64.) For reference, the PRs to enable osx-arm64 as a platform include:
Anaconda.org need to recognize Windows on ARM as an "official" platform/subdir. This change needs to be made by Anaconda, and I've created ContinuumIO/anaconda-issues#12957 as a public tracking issue. (Technically speaking, anaconda.org already allows uploading and downloading of win-arm64 packages, but some of UI and API niceties aren't available until the backend server recognizes this platform.)
Package build out. This is probably the biggest and most complicated lift; to the best of my knowledge, no clear, definitive guides for how to bootstrap new platforms exist. However, both Anaconda and conda-forge have bootstrapped new platforms fairly recently, and I would suggest reaching out to both. (Anaconda via the "Partnerships" option at https://www.anaconda.com/contact; and conda-forge at https://github.com/conda-forge/conda-forge.github.io/issues)

Answer 4 · 2022-06-07T07:33:59.000Z

Thanks for taking time to detail steps needed, that's very helpful.

For now, arch is reported as ARM64. Is that fine for conda to detect it?

Python 3.10.3 (tags/v3.10.3:a342a49, Mar 16 2022, 12:16:06) [MSC v.1929 64 bit (ARM64)] on win32
>>> sys.platform
'win32'
>>> platform.machine()
'ARM64'
>>> platform.architecture()
('64bit', 'WindowsPE')

Thank for your creating this issue. We didn't know precisely where to ask.
We'll do for sure once this architecture is recognized!

Answer 5 · 2022-06-07T14:30:08.000Z

For now, arch is reported as ARM64. Is that fine for conda to detect it?

Yes; we just need to know if Python on Windows was going to use the macOS name (arm64), the Linux name (aarch64), or something else when reporting the machine name/CPU architecture.

Answer 6 · 2022-06-07T16:06:39.000Z

@chenghlee Both of us are working on this: @pbo-linaro, @nsait-linaro

Answer 7 · 2023-04-17T17:53:47.000Z

cc @zooba (for awareness)

Answer 8 · 2023-04-17T17:58:09.000Z

@zooba Can you answer if the values in #11472 (comment) are the ones in current builds of Python for arm64?

Answer 9 · 2023-04-17T19:15:58.000Z

Yes. However, the two main detections we need are:

sys.winver.endswith("-arm64") to detect an ARM64 build of CPython
platform.processor() == "ARM64" to detect an ARM64 host OS

Because an ARM64 host can run x86 and x64 builds, just checking the processor isn't enough. But only an ARM64 host can run an ARM64 build, so if winver checks out then we're definitely on ARM64.

Also, the processor() call on 3.11 can be tricked by compatibility modes (it uses environment variables - in 3.12 it'll be more accurate). But if the winver check comes first, it won't matter.

platform.architecture() doesn't have an API that can return ARM64 - it's specified to return the number of bits in a pointer, which is 64.

Also, there's no standard way to specify that a build should be cross-compiled for ARM64. We get to invent one here (the variable used for setuptools is the one set by vcvarsall.bat, which may/may not be appropriate for conda).

Answer 10 · 2023-07-26T17:16:28.000Z

We're just removing the milestone for now, while we're working through the legal parts of this work.

Answer 11 · 2024-04-08T20:00:23.000Z

Can someone kindly provide an outlook/availability for this issue of conda supported/beta on Windows ARM?

Thank You

Answer 12 · 2024-05-27T11:53:46.000Z

With Microsoft's and Qualcomm's recent push for Arm64 Windows PCs, it might be interesting to reconsider this issue.

Microsoft Surface and our OEM partners Acer, ASUS, Dell, HP, Lenovo and Samsung, with [Snapdragon X] pre-orders beginning today and availability starting on June 18.

Answer 13 · 2024-05-28T09:59:11.000Z

A regular x64 conda will run fine on Windows ARM64, however, it will currently only install emulated binaries. conda itself can run fine on a native ARM64 Python (last time we tested it, anyway), so there's nothing to worry about there. It's up to conda-forge right now to add build support for every package to get a truly native ARM64 environment - best to show your interest over here.

I'm not sure what the "legal parts" Jannis mentioned in his last post refer to (unless it's the CLA?), so there may be some other things to do here. But this is not the project that needs the encouragement.

Answer 14 · 2024-06-19T19:34:34.000Z

I use the linux subsystem, which is not supported in that there is no miniconda. The above comment played a part in my decision to be one of the first to own a new Surface, but the x86 version would not load. So I have this wonderful machine now sort of useless. Hoping...

Answer 15 · 2024-06-19T19:41:00.000Z

If you're in WSL, then you're in Linux. You'll need an aarch64 build for Linux (probably, I haven't used WSL on ARM64 hardware, but I assume that's how it works).

The x86/x64 emulation I refer to above is a Windows feature, not a Linux feature.

Answer 16 · 2024-06-25T19:52:12.000Z

A regular x64 conda will run fine on Windows ARM64, however, it will currently only install emulated binaries. conda itself can run fine on a native ARM64 Python (last time we tested it, anyway), so there's nothing to worry about there.

One area this isn't true for yet is packages which depend explicitly on AVX instructions since those aren't yet supported by the x64 emulation layer. A handful of packages don't perform dynamic dispatch but instead bind to the host architecture to pass on to the compiler for auto-vectorization.

Answer 17 · 2024-06-25T19:54:03.000Z

The original post mentions the lack of Windows ARM CI systems, an update is GitHub Actions now have ARM images for both Windows and Linux in public beta.

Answer 18 · 2024-06-25T20:43:12.000Z

Indeed, but for now it's only for paid plans.

However, the blog article mentions they will open up this to public repositories by end of the year:

Arm runners are available to customers on our Team and Enterprise Cloud plans. We expect to begin offering Arm runners for open source and personal accounts by the end of the year.

This is a great thing, and will definitely help to support windows-arm64.

Answer 19 · 2024-06-26T15:41:13.000Z

conda-forge has ARM64 machines already available, so if you want to help get their recipes migrated to build up to and including conda, please do. (We donated a build pool while we wait for GitHub support to open up, as they're a single organisation with good coverage, which is far more efficient than trying to do individual projects.)