enable windows arm64

Question

enable windows arm64

beckermr opened this issue 2 years ago · 42 comments

Answer 1 · 2023-04-19T17:37:12.000Z

x-refs for related ecosystem issues:

Answer 2 · 2023-04-20T16:59:26.000Z

@zooba I added a few TODOs above if you or someone else want to help.

Answer 3 · 2023-04-24T16:43:51.000Z

@finnagin is looking at the conda-build changes, and will probably have some Qs about any user visible changes (such as how best to inform conda-build it should be doing a cross-compile). Hopefully that will unblock us to start looking at recipe/feedstock changes and figuring out the best way to automatically migrate existing ones (if needed).

Answer 4 · 2023-04-24T20:57:40.000Z

@zooba, @finnagin, task 4 needs to be done to build some packages. Task 5 is needed only after python itself is built.

Answer 5 · 2023-04-25T11:51:29.000Z

Task 4 is definitely first, yeah, and part of that seems to be plugging in the arch and host_arch configs to allow running an ARM64 targeted build on AMD64 machine. That plus a few recipe updates should enable building Python - we don't actually need conda/conda-build itself to run natively (it'll be nice, but it's not essential).

Answer 6 · 2023-04-25T16:11:23.000Z

msvc_env_cmd is a deprecated function, so we need not update it to actually work. We just need it to not error out. So, a check for bits to see if it is arm64 and returning without doing anything should be enough.

Answer 7 · 2023-05-03T18:19:24.000Z

@isuruf You mentioned in the call that you got the zlib build working. Can you post how you did that for @finnagin's benefit?

Answer 8 · 2023-05-03T21:32:52.000Z

See conda-forge/zlib-feedstock#72

conda build recipe -m .ci_support/win_arm64_.yaml

Answer 9 · 2023-05-03T22:12:32.000Z

Ah neat, it's using CMake's support. I guess that means the env variables needed were already being set? Or did those get slipped in recently without me noticing?

Answer 10 · 2023-05-03T22:16:12.000Z

No, we should add support for an env variable like CMAKE_ARGS which we use in Linux/macOS. The two systems are so similar that even if cmake is not told that it is cross compiling, things just work because the correct compiler is used.

Answer 11 · 2023-05-03T22:20:09.000Z

Oh cool, I didn't see where the compiler was being picked. Just a vcvarsall call? That should work for anything with MSBuild or setuptools (if relatively up to date) too.

Answer 12 · 2023-05-03T22:22:43.000Z

Just a vcvarsall call?

Yes.

C:\Program Files\Microsoft Visual Studio\2022\Enterprise>CALL "VC\Auxiliary\Build\vcvarsamd64_arm64.bat" -vcvars_ver=14.35 10.0.22621.0

Answer 13 · 2023-05-04T20:16:16.000Z

Thanks @isuruf! This is very helpful!

Answer 14 · 2023-07-06T06:47:40.000Z

There should be a zlib package for win-arm64 now. It's untested though.

Answer 15 · 2023-07-11T19:00:03.000Z

Can someone with hardware access check the zstd package?
Download https://anaconda.org/conda-forge/zstd/1.5.2/download/win-arm64/zstd-1.5.2-h45ec737_7.conda
and run

conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda

Answer 16 · 2023-07-11T21:32:59.000Z

Hey @isuruf I can try running that for you! However, I'm currently waiting for access to the network that the machine I would use is on so I likely won't be able to do it until tomorrow.

Answer 17 · 2023-07-12T09:20:07.000Z

Can someone with hardware access check the zstd package? Download https://anaconda.org/conda-forge/zstd/1.5.2/download/win-arm64/zstd-1.5.2-h45ec737_7.conda and run
conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda

yes, I download it, but how to run it? I get the error ">conda build -t zstd-1.5.2-h45ec737_7.conda
'conda' is not recognized as an internal or external command,
operable program or batch file."

Answer 18 · 2023-07-12T10:17:31.000Z

yes, I download it, but how to run it?

You need to have conda installed and conda build available. Both of these don't exist for win-arm64 yet, so I think it will be tricky to test these artefacts natively until we've built conda & conda-build...

Answer 19 · 2023-07-12T10:25:13.000Z

Looking at conda install -n test conda conda-build and filtering out the noarch/ packages, that would mean building at least:

The following NEW packages will be INSTALLED:

  brotli-python      conda-forge/win-64::brotli-python-1.0.9-py311h12c1d0e_9
  bzip2              conda-forge/win-64::bzip2-1.0.8-h8ffe710_4
  ca-certificates    conda-forge/win-64::ca-certificates-2023.5.7-h56e8100_0
  cffi               conda-forge/win-64::cffi-1.15.1-py311h7d9ee11_3
  chardet            conda-forge/win-64::chardet-5.1.0-py311h1ea47a8_0
  conda              conda-forge/win-64::conda-23.5.0-py311h1ea47a8_1
  conda-build        conda-forge/win-64::conda-build-3.25.0-py311h1ea47a8_0
  cryptography       conda-forge/win-64::cryptography-41.0.2-py311h28e9c30_0
  libarchive         conda-forge/win-64::libarchive-3.6.2-h6f8411a_1
  libexpat           conda-forge/win-64::libexpat-2.5.0-h63175ca_1
  libffi             conda-forge/win-64::libffi-3.4.2-h8ffe710_5
  libiconv           conda-forge/win-64::libiconv-1.17-h8ffe710_0
  liblief            conda-forge/win-64::liblief-0.12.3-h63175ca_0
  libsqlite          conda-forge/win-64::libsqlite-3.42.0-hcfcfb64_0
  libxml2            conda-forge/win-64::libxml2-2.11.4-hc3477c8_0
  libzlib            conda-forge/win-64::libzlib-1.2.13-hcfcfb64_5
  lz4-c              conda-forge/win-64::lz4-c-1.9.4-hcfcfb64_0
  lzo                conda-forge/win-64::lzo-2.10-he774522_1000
  m2-msys2-runtime   conda-forge/win-64::m2-msys2-runtime-2.5.0.17080.65c939c-3
  m2-patch           conda-forge/win-64::m2-patch-2.7.5-2
  m2w64-gcc-libgfor~ conda-forge/win-64::m2w64-gcc-libgfortran-5.3.0-6
  m2w64-gcc-libs     conda-forge/win-64::m2w64-gcc-libs-5.3.0-7
  m2w64-gcc-libs-co~ conda-forge/win-64::m2w64-gcc-libs-core-5.3.0-7
  m2w64-gmp          conda-forge/win-64::m2w64-gmp-6.1.0-2
  m2w64-libwinpthre~ conda-forge/win-64::m2w64-libwinpthread-git-5.0.0.4634.697f757-2
  markupsafe         conda-forge/win-64::markupsafe-2.1.3-py311ha68e1ae_0
  menuinst           conda-forge/win-64::menuinst-1.4.19-py311h1ea47a8_1
  msys2-conda-epoch  conda-forge/win-64::msys2-conda-epoch-20160418-1
  openssl            conda-forge/win-64::openssl-3.1.1-hcfcfb64_1
  psutil             conda-forge/win-64::psutil-5.9.5-py311ha68e1ae_0
  py-lief            conda-forge/win-64::py-lief-0.12.3-py311h12c1d0e_0
  pycosat            conda-forge/win-64::pycosat-0.6.4-py311ha68e1ae_1
  python             conda-forge/win-64::python-3.11.4-h2628c8c_0_cpython
  python-libarchive~ conda-forge/win-64::python-libarchive-c-4.0-py311h1ea47a8_2
  python_abi         conda-forge/win-64::python_abi-3.11-3_cp311
  pyyaml             conda-forge/win-64::pyyaml-6.0-py311ha68e1ae_5
  ripgrep            conda-forge/win-64::ripgrep-13.0.0-h7f3b576_2
  ruamel.yaml        conda-forge/win-64::ruamel.yaml-0.17.32-py311ha68e1ae_0
  ruamel.yaml.clib   conda-forge/win-64::ruamel.yaml.clib-0.2.7-py311ha68e1ae_1
  tk                 conda-forge/win-64::tk-8.6.12-h8ffe710_0
  ucrt               conda-forge/win-64::ucrt-10.0.22621.0-h57928b3_0
  vc                 conda-forge/win-64::vc-14.3-h64f974e_17                           ✅ exists for win-arm64
  vc14_runtime       conda-forge/win-64::vc14_runtime-14.36.32532-hfdfe4a8_17          ✅ exists for win-arm64
  vs2015_runtime     conda-forge/win-64::vs2015_runtime-14.36.32532-h05e6639_17        ✅ exists for win-arm64
  xz                 conda-forge/win-64::xz-5.2.6-h8d14728_0
  yaml               conda-forge/win-64::yaml-0.2.5-h8ffe710_2
  zstandard          conda-forge/win-64::zstandard-0.19.0-py311he5d195f_2
  zstd               conda-forge/win-64::zstd-1.5.2-h12be248_7                         ✅ exists for win-arm64

I think it should be possible to stub out m2-patch (at least as long as we're not applying patches, which isn't necessary for testing artefacts), so we avoid having to build the msys2-stack

Answer 20 · 2023-07-12T11:53:38.000Z

Just install an x86_64 version of conda and conda-build (and windows 11 for x86_64 emulation) and run the command.

Answer 21 · 2023-07-12T17:05:49.000Z

@isuruf when I run conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda I get the following error:

Solving environment: ...working... failed
failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}
WARNING:conda_build.build:failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}

looks like cmake and pkg-config need arm64 versions first?

Answer 22 · 2023-07-13T02:56:30.000Z

@isuruf when I run conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda I get the following error:
Solving environment: ...working... failed
failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}
WARNING:conda_build.build:failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}
looks like cmake and pkg-config need arm64 versions first?

(base) C:\Users\shijunz\Downloads>conda build -t zstd-1.5.2-h45ec737_7.conda
TEST START: zstd-1.5.2-h45ec737_7.conda
Copying package to conda-build croot. No packages otherwise alongside yours will be available unless you specify -c local.
To avoid this warning, your package must reside in a channel structure with platform-subfolders. See more info on what a
valid channel is at https://conda.io/docs/user-guide/tasks/create-custom-channels.html
WARNING:conda_build.build:Copying package to conda-build croot. No packages otherwise alongside yours will be available un
less you specify -c local. To avoid this warning, your package must reside in a channel structure with platform-subfolders
. See more info on what a valid channel is at https://conda.io/docs/user-guide/tasks/create-custom-channels.html
INFO:conda_index.index.convert_cache:Migrate database
INFO:conda_index.index.convert_cache:CONVERT .cache
INFO:conda_index.index.convert_cache:Migrate database
INFO:conda_index.index:noarch cached 0 B from 0 packages at 0 B/second
INFO:conda_index.index.convert_cache:Migrate database
INFO:conda_index.index.convert_cache:CONVERT .cache
INFO:conda_index.index.convert_cache:Migrate database
INFO:conda_index.index:win-arm64 cached 253 KB from 1 packages at 22.5 MB/second
INFO:conda_index.index:Subdir: noarch Gathering repodata
INFO:conda_index.index:noarch Writing pre-patch repodata
INFO:conda_index.index:noarch Applying patch instructions
INFO:conda_index.index:noarch Writing patched repodata
INFO:conda_index.index:noarch Building current_repodata subset
INFO:conda_index.index:noarch Writing current_repodata subset
INFO:conda_index.index:noarch Writing index HTML
INFO:conda_index.index:Completed noarch
INFO:conda_index.index:Subdir: win-arm64 Gathering repodata
INFO:conda_index.index:win-arm64 Writing pre-patch repodata
INFO:conda_index.index:win-arm64 Applying patch instructions
INFO:conda_index.index:win-arm64 Writing patched repodata
INFO:conda_index.index:win-arm64 Building current_repodata subset
INFO:conda_index.index:win-arm64 Writing current_repodata subset
INFO:conda_index.index:win-arm64 Writing index HTML
INFO:conda_index.index:Completed win-arm64
INFO:conda_index.index:Channeldata subdir: noarch
INFO:conda_index.index:Channeldata subdir: win-arm64
Multiple meta files found. The meta.yaml file in the base directory (C:\Users\shijunz\AppData\Local\Temp\tmpgcv9dcgo\info\r
ecipe) will be used.
WARNING:conda_build.utils:Multiple meta files found. The meta.yaml file in the base directory (C:\Users\shijunz\AppData\Loc
al\Temp\tmpgcv9dcgo\info\recipe) will be used.
No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.22
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of
1.22
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', "libz
lib[version='>=1.2.13,<1.3.0a0']", 'pkg-config', "vc14_runtime[version='>=14.36.32532']", "vc[version='>=14.3,<15']", 'vs20
22_win-arm64'}
WARNING:conda_build.build:failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform
win-arm64: {'cmake', "libzlib[version='>=1.2.13,<1.3.0a0']", 'pkg-config', "vc14_runtime[version='>=14.36.32532']", "vc[ver
sion='>=14.3,<15']", 'vs2022_win-arm64'}

Leaving build/test directories:
Work:
C:\ProgramData\miniconda3\conda-bld\work
Test:
C:\ProgramData\miniconda3\conda-bld\test_tmp
Leaving build/test environments:
Test:
source activate C:\ProgramData\miniconda3\conda-bld_test_env
Build:
source activate C:\ProgramData\miniconda3\conda-bld_build_env

TESTS FAILED: zstd-1.5.2-h45ec737_7.tar.bz2

(base) C:\Users\shijunz\Downloads>

Answer 23 · 2023-07-13T03:04:31.000Z

@isuruf when I run conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda I get the following error:
Solving environment: ...working... failed
failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}
WARNING:conda_build.build:failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}
looks like cmake and pkg-config need arm64 versions first?

cmake should be ok, I build a lot windows on arm64 app using the x64 version cmake. it support crossing build. maybe we should not check the it? or just tell the cmake -A arm64.

but the pkg-config I have no idea.

Answer 24 · 2023-07-13T03:19:58.000Z

Just install an x86_64 version of conda and conda-build (and windows 11 for x86_64 emulation) and run the command.

llecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'pkg-config', "vc[version='>=14.3,<15']", "vc14_runtime[version='>=14.36.32532']", 'cmake', "libzlib[version='>=1.2.13,<1.3.0a0']", 'vs2022_win-arm64'}
WARNING:conda_build.build:failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'pkg-config', "vc[version='>=14.3,<15']", "vc14_runtime[version='>=14.36.32532']", 'cmake', "libzlib[version='>=1.2.13,<1.3.0a0']", 'vs2022_win-arm64'}

yes, I run it on the windows 11 for x86-64 emulation mode) and get the similar error.

Answer 25 · 2023-07-13T03:20:25.000Z

cmake should be ok, I build a lot windows on arm64 app using the x64 version cmake. it support crossing build.

If it's relevant, in addition to cross-compiling with the x64 version cmake also has arm64 Windows releases.

Answer 26 · 2023-07-13T03:24:10.000Z

We'll need to disable those tests for now. In the meantime, can you do,

set CONDA_SUBDIR=win-arm64
conda create -n arm64 zstd
set CONDA_SUBDIR=
conda activate arm64
conda config --env --set subdir win-arm64

and then try zstd --version?

Answer 27 · 2023-07-13T03:44:38.000Z

Sure!

>zstd --version
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***

Answer 28 · 2023-07-13T06:09:04.000Z

(arm64) C:\Windows\System32>conda install zstd
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

zstd

Current channels:

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

Answer 29 · 2023-07-13T15:18:14.000Z

@shijunz, you need to add conda-forge to the list of channels.
@finnagin, thanks.

Answer 30 · 2023-07-14T00:25:32.000Z

@shijunz, you need to add conda-forge to the list of channels. @finnagin, thanks.

(env_win-arm64) C:>conda install -c conda-forge zstd
Collecting package metadata (current_repodata.json): done
Solving environment: done

Package Plan

environment location: C:\Users\shijunz\AppData\Local\miniconda3\envs\env_win-arm64

added / updated specs:
- zstd

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
libzlib-1.2.13             |       h50ba5a6_5          52 KB  conda-forge
vc-14.3                    |      h6ed9334_17          16 KB  conda-forge
vc14_runtime-14.36.32532   |      he64ce00_17        14.4 MB  conda-forge
zstd-1.5.2                 |       h45ec737_7         253 KB  conda-forge
------------------------------------------------------------
                                       Total:        14.7 MB

The following NEW packages will be INSTALLED:

libzlib conda-forge/win-arm64::libzlib-1.2.13-h50ba5a6_5
vc conda-forge/win-arm64::vc-14.3-h6ed9334_17
vc14_runtime conda-forge/win-arm64::vc14_runtime-14.36.32532-he64ce00_17
zstd conda-forge/win-arm64::zstd-1.5.2-h45ec737_7

Proceed ([y]/n)? y

Downloading and Extracting Packages

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Answer 31 · 2023-07-14T00:33:27.000Z

but, in window on arm, I can NOT run ' conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda', because I try to

(env_win-arm64) C:\Users\shijunz\Downloads>conda install -c conda-forge conda-build
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

conda-build

Current channels:

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

Answer 32 · 2023-07-14T00:43:42.000Z

I run conda install conda-build at the x64 env, and conda info show the conda-build is ok, but run : conda build -t zstd-1.5.2-h45ec737_7.tar.bz2n and get the same error.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', "vc14_runtime[version='>=14.36.32532']", 'pkg-config', "libzlib[version='>=1.2.13,<1.3.0a0']", "vc[version='>=14.3,<15']", 'vs2022_win-arm64'}
WARNING:conda_build.build:failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', "vc14_runtime[version='>=14.36.32532']", 'pkg-config', "libzlib[version='>=1.2.13,<1.3.0a0']", "vc[version='>=14.3,<15']", 'vs2022_win-arm64'}

Answer 33 · 2024-02-16T14:29:50.000Z

win_arm64 should be added to the provider section of the docs

Answer 34 · 2024-05-04T07:03:42.000Z

I'm going to unpin this for now to make space for the numpy migration in the top bar, which is going to hit a lot of feedstocks now. We can repin it of course once things calm down there.

Answer 35 · 2024-07-09T23:43:20.000Z

Repinning this now that a slot opened up again, after the stdlib migration has been unpinned

Answer 36 · 2024-07-20T02:04:41.000Z

cc @arnabanimesh (as this context may also be useful in your efforts to rebuild for Windows ARM 🙂)