aarch64: numpy compile is failing
belm0 opened this issue · 11 comments
(aarch64, python3.8) appears to map to numpy==1.19.2, but compile from source of that numpy version fails on aarch64
We know that recent numpy versions (e.g. numpy==1.24.2) build fine on aarch64. So there must be a more suitable oldest-supported-numpy than 1.19.2.
This is for building numpy-quaternion for an accelerated python implementation (pyston). (There's no problem building this package from source on amd64.)
Collecting numpy-quaternion==2022.4.3 (from -r base_adjusted.txt (line 83))
Downloading numpy-quaternion-2022.4.3.tar.gz (60 kB)
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [89 lines of output]
Looking in indexes: https://pypi.org/simple
Collecting setuptools!=50.0
Using cached setuptools-68.2.2-py3-none-any.whl.metadata (6.3 kB)
Collecting wheel
Using cached wheel-0.41.2-py3-none-any.whl.metadata (2.2 kB)
Collecting oldest-supported-numpy
Downloading oldest_supported_numpy-2023.8.3-py3-none-any.whl.metadata (9.5 kB)
Collecting numpy==1.19.2 (from oldest-supported-numpy)
Downloading numpy-1.19.2.zip (7.3 MB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [57 lines of output]
Error compiling Cython file:
------------------------------------------------------------
...
for i in range(1, RK_STATE_LEN):
self.rng_state.key[i] = val[i]
self.rng_state.pos = i
self._bitgen.state = &self.rng_state
self._bitgen.next_uint64 = &mt19937_uint64
^
------------------------------------------------------------
_mt19937.pyx:138:35: Cannot assign type 'uint64_t (*)(void *) except? -1 nogil' to 'uint64_t (*)(void *) noexcept nogil'. Exception values are incompatible. Suggest adding 'noexcept' to type 'uint64_t (void *) except? -1 nogil'.
This is a question of numpy at the time not having a cap on the required cython version, and not being compatible with Cython 3 that got released years later. You just have to add cython <3 to your dependencies and then things should work.
You just have to add cython <3 to your dependencies and then things should work.
This is about pip's build of numpy-quaternion. What would I add that cython dependency to?
This is a question of numpy at the time not having a cap on the required cython version, and not being compatible with Cython 3 that got released years later.
It sounds like numpy 1.19.2 is not fit then, and oldest-supported-numpy should map to something that got the cython version correct.
We could decide to universally add cython <3, as a requirement, as all the pinned python versions we have here don't support Cython 3 yet.
What would I add that cython dependency to?
To whatever environment where you're trying to build numpy yourself.
We could decide to universally add cython <3, as a requirement, as all the pinned python versions we have here don't support Cython 3 yet.
This sounds good.
extensive discussion here: https://discuss.python.org/t/no-way-to-pin-build-dependencies
The workaround is to use the PIP_CONSTRAINT environment variable (-c option doesn't work, because it doesn't apply to build environments).
echo 'cython < 3' > c.txt
PIP_CONSTRAINT=c.txt pip ...Actually, the cython < 3 workaround doesn't work for my case, because I'm using pip wheel with multiple packages, and some of them need cython 3.
Collecting wsaccel==0.6.6 (from -r base_adjusted.txt (line 152))
Downloading wsaccel-0.6.6.tar.gz (289 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 289.7/289.7 kB 17.2 MB/s eta 0:00:00
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [13 lines of output]
Looking in indexes: https://pypi.org/simple,
ERROR: Cannot install Cython~=3.0.0 because these package versions have conflicting dependencies.
The conflict is caused by:
The user requested Cython~=3.0.0
The user requested (constraint) cython<3
We could decide to universally add cython <3, as a requirement, as all the pinned python versions we have here don't support Cython 3 yet.
We now have Python 3.12 / NumPy 1.26 which does support Cython 3.0. But we could still add:
cython<3.0; python_version<'3.12'However, the package using oldest-supported-numpy may not need Cython at all, or need Cython 3.x - and then we're breaking something that used to work, or adding an unnecessary dependency. The vast majority of oldest-supported-numpy usage will be pulling in numpy wheels, for which Cython isn't relevant.
Actually, the
cython < 3workaround doesn't work for my case, because I'm usingpip wheelwith multiple packages, and some of them need cython 3.
I think if there's conflicting constraints, there is no way to do anything to build all packages in a single env. You can split the build (pip wheel or any other such command) to do that per-package or per group of packages.
(aarch64, python3.8) appears to map to numpy==1.19.2, but compile from source of that numpy version fails on aarch64
That is the correct version, there are manylinux wheels at least: https://pypi.org/project/numpy/1.19.2/#files. Which is the main thing that this package cares about - we always target the oldest numpy version for which there are wheels.
I assume you worked around the problem by now @belm0? I think we can close this issue, because I don't see what change we can safely make here.
👋 I'm hitting this as well, it seems that numpy 1.19.3 fixes this by adding "Cython>=0.29.21,<3.0",
https://github.com/numpy/numpy/blob/v1.19.3/pyproject.toml#L6
I don't know all the implication but could we update the requirements here to 1.19.3 ?
Thanks for pointing that out @vincentsarago. That should be a safe change to make.
Fix merged in gh-80, and new release is up: https://pypi.org/project/oldest-supported-numpy/2023.12.21/. Thanks all!