numpy 1.20* inconsistent in recognising CPU features

Question

numpy 1.20* inconsistent in recognising CPU features

dimpase opened this issue 4 years ago · 62 comments

typically this looks like the following:
https://groups.google.com/d/msgid/sage-devel/763f8650-9803-4bba-a0ec-46744204fc22n%40googlegroups.com?utm_medium=email&utm_source=footer

[sagelib-9.4.beta2]   File "/home/chapoton/sage/local/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module>
[sagelib-9.4.beta2]     from numpy.core._multiarray_umath import (
[sagelib-9.4.beta2] RuntimeError: NumPy was built with baseline optimizations: 
[sagelib-9.4.beta2] (SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42) but your machine doesn't support:
[sagelib-9.4.beta2] (POPCNT).

CC: @kliem @dcoudert @fchapoton

Component: build

Author: Jonathan Kliem

Branch/Commit: a7eef1b

Reviewer: John Palmieri, Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/32021

Answer 1 · 2021-06-21T12:45:40.000Z

comment:2

See also https://groups.google.com/g/sage-release/c/lmUFq4d7is0/m/uKRsE9KoBQAJ

It could be that builds in question actually took a prebuilt on another machine numpy, then it's a numpy bug, I guess, to install an incompatible binary.

Answer 2 · 2021-06-21T22:21:14.000Z

comment:5

I see this when building from scratch on an iMac Pro, OS X 10.15.7, with various homebrew packages installed. numpy is built from scratch and says

########### EXT COMPILER OPTIMIZATION ###########
Platform      : 
  Architecture: x64
  Compiler    : gcc

CPU baseline  : 
  Requested   : 'native'
  Enabled     : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_SKX
  Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mavx512cd -mavx512vl -mavx512bw -mavx512dq
  Extra checks: AVX512F_REDUCE AVX512BW_MASK AVX512DQ_MASK

CPU dispatch  : 
  Requested   : 'max -xop -fma4'
  Enabled     : AVX512_KNL AVX512_CLX AVX512_CNL AVX512_ICL
  Generated   : none
CCompilerOpt._cache_write[796] : write cache to path -> /Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta2/local/var/tmp/sage/build/numpy-1.20.3/src/build/temp.macosx-10.15-x86_64-3.9/ccompiler_opt_cache_ext.py

########### CLIB COMPILER OPTIMIZATION ###########
Platform      : 
  Architecture: x64
  Compiler    : gcc

CPU baseline  : 
  Requested   : 'native'
  Enabled     : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_SKX
  Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mavx512cd -mavx512vl -mavx512bw -mavx512dq
  Extra checks: AVX512F_REDUCE AVX512BW_MASK AVX512DQ_MASK

CPU dispatch  : 
  Requested   : 'max -xop -fma4'
  Enabled     : AVX512_KNL AVX512_CLX AVX512_CNL AVX512_ICL
  Generated   : none

scipy fails with

    RuntimeError: NumPy was built with baseline optimizations:
    (SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_SKX) but your machine doesn't support:
    (AVX512F).

I notice that we deleted related numpy patches in the latest update, presumably because they were included into the source code. There must have been other changes in the source that broke something else.

Answer 3 · 2021-06-21T22:33:14.000Z

comment:6

I cannot build Sage on this machine right now; does that this ticket a blocker, or is it too architecture-specific? (I can build on some other OS X machines just fine.)

Answer 4 · 2021-06-22T11:32:05.000Z

comment:7

If you add --cpu-baseline="sse2" to `spkg-install. Will this work? Similar to #31565.

Answer 5 · 2021-06-22T12:36:33.000Z

comment:8

Replying to @jhpalmieri:

I cannot build Sage on this machine right now; does that this ticket a blocker, or is it too architecture-specific? (I can build on some other OS X machines just fine.)

Can you do a standalone build of scipy+numpy on that machine?
Or does it fail in the same way?

Answer 6 · 2021-06-22T15:16:20.000Z

comment:9

Replying to @dimpase:

Replying to @jhpalmieri:

I cannot build Sage on this machine right now; does that this ticket a blocker, or is it too architecture-specific? (I can build on some other OS X machines just fine.)

Can you do a standalone build of scipy+numpy on that machine?
Or does it fail in the same way?

I can't easily mimic the Sage install. If I just unpack the zip file for numpy and run python3 setup.py build -j 4 install --prefix $HOME/.local, it complains about blas stuff:

RuntimeError: Found /usr/lib/libcblas.dylib, but that file is a symbolic link to the MacOS Accelerate framework, which is not supported by NumPy. You must configure the build to use a different optimized library, or disable the use of optimized BLAS and LAPACK by setting the environment variables NPY_BLAS_ORDER="" and NPY_LAPACK_ORDER="" before building NumPy.

I tried running Sage's lapack_conf.py, but it complained about not finding cblas. numpy installed after export NPY_BLAS_ORDER="" and export NPY_LAPACK_ORDER="", but scipy did not.

So I don't understand how to build numpy+scipy on this machine.

Answer 7 · 2021-06-22T15:27:19.000Z

comment:10

Replying to @kliem:

If you add --cpu-baseline="sse2" to `spkg-install. Will this work? Similar to #31565.

I tried

diff --git a/build/pkgs/numpy/spkg-install.in b/build/pkgs/numpy/spkg-install.in
index 1237d059d3..5d45f176ae 100644
--- a/build/pkgs/numpy/spkg-install.in
+++ b/build/pkgs/numpy/spkg-install.in
@@ -22,7 +22,7 @@ python3 ../lapack_conf.py
 export FFLAGS="$FFLAGS -fPIC"
 export FCFLAGS="$FCFLAGS -fPIC"
 
-export NUMPY_FCONFIG="config_fc --noopt --noarch"
+export NUMPY_FCONFIG="config_fc --noopt --noarch --cpu-baseline=\"sse2\""
 
 ################################################

but this isn't the right syntax:

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --cpu-baseline not recognized

Can you provide more guidance?

Answer 8 · 2021-06-22T18:22:19.000Z

comment:11

Replying to @jhpalmieri:

Replying to @kliem:

If you add --cpu-baseline="sse2" to `spkg-install. Will this work? Similar to #31565.

I tried

diff --git a/build/pkgs/numpy/spkg-install.in b/build/pkgs/numpy/spkg-install.in
index 1237d059d3..5d45f176ae 100644
--- a/build/pkgs/numpy/spkg-install.in
+++ b/build/pkgs/numpy/spkg-install.in
@@ -22,7 +22,7 @@ python3 ../lapack_conf.py
 export FFLAGS="$FFLAGS -fPIC"
 export FCFLAGS="$FCFLAGS -fPIC"
 
-export NUMPY_FCONFIG="config_fc --noopt --noarch"
+export NUMPY_FCONFIG="config_fc --noopt --noarch --cpu-baseline=\"sse2\""
 
 ################################################

but this isn't the right syntax:

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --cpu-baseline not recognized

Can you provide more guidance?

Thanks for catching this. Actually you need to do it during a different command:

diff --git a/build/pkgs/numpy/spkg-install.in b/build/pkgs/numpy/spkg-install.in
index 1237d059d3..c261d89444 100644
--- a/build/pkgs/numpy/spkg-install.in
+++ b/build/pkgs/numpy/spkg-install.in
@@ -28,4 +28,4 @@ export NUMPY_FCONFIG="config_fc --noopt --noarch"
 
 sdh_setup_bdist_wheel ${NUMPY_FCONFIG}
 
-sdh_store_and_pip_install_wheel .
+sdh_store_and_pip_install_wheel . --cpu-baseline="sse2"

A cleaner solution and also a good solution for a branch might be to disable -march=native via

export CFLAGS="$CFLAGS_NON_NATIVE"

Numpy will still compile intrinsics to be dispatched on runtime. (And even if it isn't the case, the policy was that -march=native should only be enabled as long as this does not effect portability). Either way it appears to be a numpy bug, because it detects your architecture wrongly.

I hope one of those things work.

Edited: See comment below

Answer 9 · 2021-06-22T18:23:03.000Z

comment:12

Actually you don't want cpu-dispatch but cpu-baseline as before.

Answer 10 · 2021-06-22T18:42:56.000Z

comment:13

Replying to @kliem:

Thanks for catching this. Actually you need to do it during a different command:

diff --git a/build/pkgs/numpy/spkg-install.in b/build/pkgs/numpy/spkg-install.in
index 1237d059d3..c261d89444 100644
--- a/build/pkgs/numpy/spkg-install.in
+++ b/build/pkgs/numpy/spkg-install.in
@@ -28,4 +28,4 @@ export NUMPY_FCONFIG="config_fc --noopt --noarch"
 
 sdh_setup_bdist_wheel ${NUMPY_FCONFIG}
 
-sdh_store_and_pip_install_wheel .
+sdh_store_and_pip_install_wheel . --cpu-baseline="sse2"

This doesn't work, unfortunately:

Error: sdh_store_wheel requires .  as only argument

Can we build numpy using other of Sage's helpers which accept arguments?

A cleaner solution and also a good solution for a branch might be to disable -march=native via
export CFLAGS="$CFLAGS_NON_NATIVE"

I tried various options including

diff --git a/build/pkgs/numpy/spkg-install.in b/build/pkgs/numpy/spkg-install.in
index 1237d059d3..88aee50f6a 100644
--- a/build/pkgs/numpy/spkg-install.in
+++ b/build/pkgs/numpy/spkg-install.in
@@ -6,6 +6,9 @@ if [ `uname` = "Darwin" ]; then
     unset ATLAS
     unset BLAS
     unset LAPACK
+    export CFLAGS="$CFLAGS_NON_NATIVE"
+    export FCFLAGS="$FCFLAGS_NON_NATIVE"
+    export FFLAGS="$FFLAGS_NON_NATIVE"
 else
     export {ATLAS,PTATLAS,OPENBLAS,MKL,MKLROOT}=None
     export LDFLAGS="${LDFLAGS} -shared"

but they didn't change the "COMPILER OPTIMIZATION" summaries when building numpy and they did not allow scipy to build.

Answer 11 · 2021-06-22T19:52:56.000Z

comment:14

You are exactly right. The previous place was the correct place to use it.

export NUMPY_FCONFIG="config_fc --noopt --noarch --cpu-baseline=\"sse2\""

However:

The command arguments are available in build, build_clib, and build_ext. if build_clib or build_ext are not specified by the user, the arguments of build will be used instead, which also holds the default values.

So this does not work for bdist_wheel and this is what we use. Apparently, when build with wheel it just gets optimized regardless?

Answer 12 · 2021-06-22T20:41:00.000Z

comment:15

Replying to @jhpalmieri:

Replying to @kliem:

Thanks for catching this. Actually you need to do it during a different command:

diff --git a/build/pkgs/numpy/spkg-install.in b/build/pkgs/numpy/spkg-install.in
index 1237d059d3..c261d89444 100644
--- a/build/pkgs/numpy/spkg-install.in
+++ b/build/pkgs/numpy/spkg-install.in
@@ -28,4 +28,4 @@ export NUMPY_FCONFIG="config_fc --noopt --noarch"
 
 sdh_setup_bdist_wheel ${NUMPY_FCONFIG}
 
-sdh_store_and_pip_install_wheel .
+sdh_store_and_pip_install_wheel . --cpu-baseline="sse2"

This doesn't work, unfortunately:

Error: sdh_store_wheel requires .  as only argument

Can we build numpy using other of Sage's helpers which accept arguments?

A cleaner solution and also a good solution for a branch might be to disable -march=native via
export CFLAGS="$CFLAGS_NON_NATIVE"

I tried various options including

diff --git a/build/pkgs/numpy/spkg-install.in b/build/pkgs/numpy/spkg-install.in
index 1237d059d3..88aee50f6a 100644
--- a/build/pkgs/numpy/spkg-install.in
+++ b/build/pkgs/numpy/spkg-install.in
@@ -6,6 +6,9 @@ if [ `uname` = "Darwin" ]; then
     unset ATLAS
     unset BLAS
     unset LAPACK
+    export CFLAGS="$CFLAGS_NON_NATIVE"
+    export FCFLAGS="$FCFLAGS_NON_NATIVE"
+    export FFLAGS="$FFLAGS_NON_NATIVE"
 else
     export {ATLAS,PTATLAS,OPENBLAS,MKL,MKLROOT}=None
     export LDFLAGS="${LDFLAGS} -shared"

but they didn't change the "COMPILER OPTIMIZATION" summaries when building numpy and they did not allow scipy to build.

As Matthias Köppe pointed out you can do the following change:

diff --git a/build/pkgs/numpy/spkg-install.in b/build/pkgs/numpy/spkg-install.in
index 1237d059d3..ef2a322ecb 100644
--- a/build/pkgs/numpy/spkg-install.in
+++ b/build/pkgs/numpy/spkg-install.in
@@ -26,6 +26,6 @@ export NUMPY_FCONFIG="config_fc --noopt --noarch"
 
 ################################################
 
-sdh_setup_bdist_wheel ${NUMPY_FCONFIG}
+sdh_setup_bdist_wheel build ${NUMPY_FCONFIG}
 
 sdh_store_and_pip_install_wheel .

Once you have done this, you can add all the previous options (added to NUMPY_FCONFIG):

cpu-baseline=min,
cpu-dispatch
try again export CFLAGS=CFLAGS_NON_NATIVE.

Actually you can cancel the numpy built once it starts compiling to immediatly see the optimization options as in comment:5

Answer 13 · 2021-06-22T22:12:31.000Z

comment:16

Sorry, I'm still not getting it. I made the change suggested in comment:15, adding "build" to the command. If I use

export NUMPY_FCONFIG="config_fc --noopt --noarch --cpu-baseline=\"sse2\""

I still get "error: option --cpu-baseline not recognized". Same with --cpu-dispatch if it's at the end. If I instead use

export NUMPY_FCONFIG="--cpu-baseline=\"sse2\" config_fc --noopt --noarch"

then numpy builds but with no apparent difference to the outcome, and scipy still fails. This happens whether or not I include CFLAGS="$CFLAGS_NON_NATIVE". Note that no matter what I try, the following appears several times in the log file:

CCompilerOpt.__init__[2103] : native flag is specified through environment variables. force cpu-baseline='native'

I don't know if that's relevant.

Answer 14 · 2021-06-23T02:02:30.000Z

comment:17

This worked for me, until we find something better:

diff --git a/build/pkgs/numpy/spkg-install.in b/build/pkgs/numpy/spkg-install.in
index 1237d059d3..50aa1529b3 100644
--- a/build/pkgs/numpy/spkg-install.in
+++ b/build/pkgs/numpy/spkg-install.in
@@ -22,6 +22,8 @@ python3 ../lapack_conf.py
 export FFLAGS="$FFLAGS -fPIC"
 export FCFLAGS="$FCFLAGS -fPIC"
 
+export CFLAGS=" -march=skylake"
+
 export NUMPY_FCONFIG="config_fc --noopt --noarch"
 
 ################################################

Answer 15 · 2021-06-23T07:44:25.000Z

comment:18

numpy/numpy#19067

Answer 16 · 2021-06-23T08:03:18.000Z

comment:19

Replying to @sheerluck:

numpy/numpy#19067

good catch!

John, does the patch from there,
https://patch-diff.githubusercontent.com/raw/numpy/numpy/pull/19074.diff
fix your issue?

Answer 17 · 2021-06-23T18:12:07.000Z

comment:20

Replying to @dimpase:

Replying to @sheerluck:

numpy/numpy#19067

good catch!

John, does the patch from there,
https://patch-diff.githubusercontent.com/raw/numpy/numpy/pull/19074.diff
fix your issue?

Unfortunately not. I see the same summary as in comment:5, and scipy fails to build.

Answer 18 · 2021-06-23T18:20:30.000Z

comment:21

By the way, where is CFLAGS_NON_NATIVE set? If I put echo $CFLAGS and echo $CFLAGS_NON_NATIVE in spkg-install.in, they both print the same thing:

-O2 -g -march=native

I'm not sure what "NON_NATIVE" means, but is it odd that it includes -march=native? If I explicitly do export CFLAGS="-O2 -g", then scipy builds.

Answer 19 · 2021-06-23T18:34:27.000Z

comment:22

Do you have default CFLAGS? User settings take precedence over those. CFLAGS_NON_NATIVE means we don't add -march=native. That's what I figured is going on by the way.

If you don't have CFLAGS set by default than this is a bug.

Answer 20 · 2021-06-23T18:40:13.000Z

comment:23

The configured flags are in sage/build/bin/sage-build-env-config.

In sage/build/bin/sage/sage-build-env we set the flags.
Priorities are in this order:

Whatever is currently set.
During configuration.
Sage's default.

CFLAGS_NON_NATIVE only is meaningful for the third one. Otherwise you choice will be set everywhere regardless (some packages might add something).

Answer 21 · 2021-06-23T20:37:23.000Z

comment:24

I inserted more echo commands. echo $CFLAGS_NON_NATIVE returns -O2 -g when I print in sage-spkg, but it returns -O2 -g -march=native when this echo command is the first thing in numpy's spkg-install.in script. That is, my numpy log file looks like:

...
Package 'numpy' is currently not installed
Uninstalling 'numpy' with legacy uninstaller
!!!!!!!!!!!!!!!!
PRINTED BY sage-spkg
cflags: -O2 -g -march=native
non-native: -O2 -g
!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!
PRINTED BY NUMPY's spkg-install.in
cflags: -O2 -g -march=native
non-native: -O2 -g -march=native
!!!!!!!!!!!!!!!!
...

After the installation is complete, one more echo command in sage-spkg returns -O2 -g again.

The problem seems to be that we (in the full script spkg-install, created from spkg-install.in) also run the script build/bin/sage-build-env, and the line

    export CFLAGS_NON_NATIVE="$ORIGINAL_CFLAGS"

produces the wrong answer.

Answer 22 · 2021-06-23T20:44:04.000Z

comment:25

One question would be, since sage-spkg already has the lines

. sage-build-env-config
. sage-build-env

why does it also need to write this into spkg-install:

for lib in "\$SAGE_ROOT/build/bin/sage-dist-helpers" "\$SAGE_SRC/bin/sage-env-config" "\$SAGE_SRC/bin/sage-env" "\$SAGE_ROOT/build/bin/sage-build-env-config" "\$SAGE_ROOT/build/bin/sage-build-env"; do
    source "\$lib"
    if [ \$? -ne 0 ]; then
        echo >&2 "Error: failed to source \$lib"
        echo >&2 "Is \$SAGE_ROOT the correct SAGE_ROOT?"
        exit 1
    fi
done

Should we omit sage-build-env-config and sage-build-env from this list?

Answer 23 · 2021-06-23T21:13:35.000Z

comment:26

I think the idea is that these wrapped scripts can also be executed manually

Answer 24 · 2021-06-23T21:16:31.000Z

comment:27

Sage builds on this machine with these changes:

diff --git a/build/bin/sage-spkg b/build/bin/sage-spkg
index c92f315f36..561e8f5cce 100755
--- a/build/bin/sage-spkg
+++ b/build/bin/sage-spkg
@@ -525,7 +525,7 @@ export PKG_NAME="$PKG_NAME"
 export PKG_BASE="$PKG_BASE"
 export PKG_VER="$PKG_VER"
 
-for lib in "\$SAGE_ROOT/build/bin/sage-dist-helpers" "\$SAGE_SRC/bin/sage-env-config" "\$SAGE_SRC/bin/sage-env" "\$SAGE_ROOT/build/bin/sage-build-env-config" "\$SAGE_ROOT/build/bin/sage-build-env"; do
+for lib in "\$SAGE_ROOT/build/bin/sage-dist-helpers" "\$SAGE_SRC/bin/sage-env-config" "\$SAGE_SRC/bin/sage-env"; do
     source "\$lib"
     if [ \$? -ne 0 ]; then
         echo >&2 "Error: failed to source \$lib"
diff --git a/build/pkgs/numpy/spkg-install.in b/build/pkgs/numpy/spkg-install.in
index 1237d059d3..0eb81094db 100644
--- a/build/pkgs/numpy/spkg-install.in
+++ b/build/pkgs/numpy/spkg-install.in
@@ -22,6 +22,8 @@ python3 ../lapack_conf.py
 export FFLAGS="$FFLAGS -fPIC"
 export FCFLAGS="$FCFLAGS -fPIC"
 
+export CFLAGS="$CFLAGS_NON_NATIVE"
+
 export NUMPY_FCONFIG="config_fc --noopt --noarch"
 
 ################################################

Answer 25 · 2021-06-23T21:17:11.000Z

comment:28

Replying to @mkoeppe:

I think the idea is that these wrapped scripts can also be executed manually

I guess they should be written so that they're idempotent.

Answer 26 · 2021-06-23T21:17:41.000Z

comment:29

Yes, that is what I was about to suggest.

Answer 27 · 2021-06-23T21:25:58.000Z

comment:30

This may be a good occasion to move all if's from sage-build-env-config to sage-build-env.

Answer 28 · 2021-06-23T21:39:44.000Z

comment:31

I fear that I may have derailed the ticket from its original purpose. The patch in comment:19 looks quite relevant to that so maybe we should include it, even though it doesn't fix the problem on my machine. Separate ticket for changes to sage-build-env and sage-build-env-config?

Answer 29 · 2021-06-23T21:45:29.000Z

Author: Jonathan Kliem

Answer 30 · 2021-06-23T21:45:29.000Z

Commit: b3e2a53

Answer 31 · 2021-06-23T21:45:29.000Z

Branch: public/32021

Answer 32 · 2021-06-23T21:45:29.000Z

comment:32

Feel free to add yourself as the author. I don't know at this point, who really is the author here.

New commits:

`07a66e0`	`header guard for sage-build-env`
`ba795b5`	`move ifs from sage-build-env-config to sage-build-env`
`b3e2a53`	`disable march=native for numpy`

Answer 33 · 2021-06-23T21:46:03.000Z

comment:33

By the way, I don't know if numpy is right or if scipy is right about my machine: maybe scipy is wrong when it says that AVX512F is not supported? How to know for sure?

Answer 34 · 2021-06-23T21:48:39.000Z

comment:34

lscpu or echo | gcc -dM -E - -march=native.

Answer 35 · 2021-06-23T21:51:07.000Z

Changed commit from b3e2a53 to 9c87b65

Answer 36 · 2021-06-23T21:51:07.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`9c87b65`	`correct explanation as popcnt wasnt the problem here`

Answer 37 · 2021-06-23T21:55:18.000Z

comment:36

If I run echo | gcc -dM -E - -march=native, it prints many things including

#define __AVX512F__ 1

OS X doesn't include lscpu, but if I run sysctl -a | grep machdep.cpu, it tells me

machdep.cpu.leaf7_features: RDWRFSGS TSC_THREAD_OFFSET BMI1 HLE AVX2 FDPEO SMEP BMI2 ERMS INVPCID RTM PQM FPU_CSDS MPX PQE AVX512F AVX512DQ RDSEED ADX SMAP CLFSOPT CLWB IPT AVX512CD AVX512BW AVX512VL MDCLEAR TSXFA IBRS STIBP L1DF SSBD

Does this mean that AVX512F is a feature, so scipy is wrong?

Answer 38 · 2021-06-23T21:58:11.000Z

comment:37

That does look like scipy is wrong.

Answer 39 · 2021-06-23T22:07:26.000Z

comment:38

The error arises when doing import numpy in Python, which scipy must try to do early in its build process. So it's not really scipy, it's within numpy. I don't know how to track this down.

Answer 40 · 2021-06-23T22:09:02.000Z

comment:39

It happens here:

https://github.com/numpy/numpy/blob/623bc1fae1d47df24e7f1e29321d0c0ba2771ce0/numpy/core/src/common/npy_cpu_features.c.src

Around line 160. For some reason this check does not work for AVX512F in your case. So yes, this is a numpy bug.

Answer 41 · 2021-06-23T22:47:27.000Z

comment:40

I opened numpy/numpy#19319.

Answer 42 · 2021-06-24T07:01:19.000Z

comment:41

With this patch, I'm able to compile on a Intel MacBook Air with macOS 10.15.7. I will run doctests asap.

Answer 43 · 2021-06-24T17:17:06.000Z

comment:42

Works for me, too, and doctests pass as well.

Answer 44 · 2021-06-28T18:58:26.000Z

comment:43

I would like someone with more familiarity with the build scripts to look over those changes. Also, there is now a NumPy bugfix for my particular problem: numpy/numpy#19362. I think that it is safest to use export CFLAGS="$CFLAGS_NON_NATIVE" for now when building NumPy and not worry about using this patch, but at some point we should investigate whether they've fixed all of the relevant problems.

Answer 45 · 2021-06-28T19:07:17.000Z

comment:44

This PR has been backported to 1.21.x as numpy/numpy#19365, but there is no indication whether it will also be backported to 1.20.x. Have you tried whether the patch applies?

Answer 46 · 2021-06-28T19:17:12.000Z

comment:45

src/bin/sage-env-config.in uses SAGE_ENV_CONFIG_SOURCED and src/bin/sage-env uses SAGE_ENV_SOURCED, so probably SAGE_HAS_RUN_SAGE_BUILD_ENV should be renamed to follow the same pattern.

Otherwise the changes to sage-build-env, sage-build-env-config.in look good to me. Thanks for simplifying the latter!

I would probably prefer to use the upstream fix, if easily possible, rather than the workaround with weaker CFLAGS.

Answer 47 · 2021-06-28T19:48:59.000Z

comment:46

The upstream fix will work for me (if it applies to 1.20.3 — I haven't tested yet), but I don't know if it will work for the issue cited in the ticket description.

Answer 48 · 2021-06-28T20:50:46.000Z

comment:47

The patch from comment:18 might work for this. As we have a working branch, the reasonable thing would be to open another ticket that tries to apply both patches to fix the problem?

Answer 49 · 2021-06-28T20:54:47.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`56ffafb`	`more consistent header guard name`

Answer 50 · 2021-06-28T20:54:47.000Z

Changed commit from 9c87b65 to 56ffafb

Answer 51 · 2021-06-28T21:32:41.000Z

comment:49

The upstream patch applies and Sage builds with it. I like the idea of opening another ticket with the two patches and also attempting to remove export CFLAGS="$CFLAGS_NON_NATIVE".

Answer 52 · 2021-06-29T00:06:42.000Z

comment:50

Replying to @jhpalmieri:

The upstream patch applies and Sage builds with it. I like the idea of opening another ticket with the two patches and also attempting to remove export CFLAGS="$CFLAGS_NON_NATIVE".

See #32080.

Answer 53 · 2021-07-06T18:21:03.000Z

Changed commit from 56ffafb to a7eef1b

Answer 54 · 2021-07-06T18:21:03.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f02780f`	`.github/workflows/ci-cygwin-*: Repair passing '--enable-fat-binary' to configure`
`a7eef1b`	`Merge tag '9.4.beta4' into t/32021/public/32021`

Answer 55 · 2021-07-06T18:21:38.000Z

comment:52

I've pushed a fix for the Cygwin CI script (see #32080).

Answer 56 · 2021-07-06T18:22:09.000Z

Reviewer: https://github.com/mkoeppe/sage/actions/runs/1005406512

Answer 57 · 2021-07-07T18:46:25.000Z

comment:54

It's all fine with me. I am willing to trust Matthias' bot for the Cygwin stuff, but I would be happier if someone else could verify the problem and solution. I've taken a look at the changes to sage-build-env and sage-build-env-config.in and they make sense, but again, I would like more people to look at it.

Given those caveats, positive review from me.

Answer 58 · 2021-07-07T18:46:25.000Z

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/1005406512 to John Palmieri, https://github.com/mkoeppe/sage/actions/runs/1005406512, ...

Answer 59 · 2021-07-07T19:33:36.000Z

comment:55

probably this should be tested with a pynac fix for Cygwin - otherwise there are errors installing pynac.

Answer 60 · 2021-07-07T20:01:50.000Z

comment:56

Looks good enough for me on Cygwin. The pynac problem is unrelated.

Let's get this in.

Answer 61 · 2021-07-07T20:01:50.000Z

Changed reviewer from John Palmieri, https://github.com/mkoeppe/sage/actions/runs/1005406512, ... to John Palmieri, Matthias Koeppe

Answer 62 · 2021-07-18T22:53:27.000Z

Changed branch from public/32021 to a7eef1b