cygwin: Try again to use system BLAS and system R

Question

cygwin: Try again to use system BLAS and system R

Closed this issue 2 years ago · 16 comments

mkoeppe commented 5 years ago

... they were disabled in spkg-configure for 9.1 in #29538 ...

Previous system BLAS tickets:

#29025, #29051, #29398

Depends on #30157
Depends on #30149
Depends on #29441
Depends on #30127
Depends on #30277

CC: @embray @slel

Component: porting: Cygwin

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/cygwin__try_again_to_use_system_blas_and_system_r @ 7f8ede4

Reviewer: github.com/mkoeppe/sage/actions/runs/207840100, github.com/mkoeppe/sage/actions/runs/207840103

Issue created by migration from https://trac.sagemath.org/ticket/30163

Answer 1 · 2020-08-14T00:08:27.000Z

Changed dependencies from #30157, #30149, #29441 to #30157, #30149, #29441, #30127, #30277

Answer 2 · 2020-08-14T00:09:02.000Z

Branch: u/mkoeppe/cygwin__try_again_to_use_system_blas_and_system_r

Answer 3 · 2020-08-14T00:14:23.000Z

Reviewer: github.com/mkoeppe/sage/actions/runs/207840100, github.com/mkoeppe/sage/actions/runs/207840103

Answer 4 · 2020-08-14T00:14:23.000Z

Author: Matthias Koeppe

Answer 5 · 2020-08-14T00:14:23.000Z

Last 10 new commits:

`b8e366e`	`build/pkgs/sagelib/spkg-src: chmod +x`
`0473ef3`	`Merge branch 't/21559/change-src-bin-installation' into t/29950/build_sagelib_using_installed_sage_setup`
`7244371`	`Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup`
`4344f89`	`Merge branch 't/21559/change-src-bin-installation' into t/29950/build_sagelib_using_installed_sage_setup`
`01b96b0`	`Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup`
`2818739`	`Merge branch 't/29950/build_sagelib_using_installed_sage_setup' into t/30277/remove_src_module_list_py`
`8a19fe2`	`build/make/Makefile.in (sagelib-clean): Clean the new build location`
`ccc67b0`	`src/sage_setup: Update cythonized_dir in doctests`
`df38027`	`Merge branch 't/29950/build_sagelib_using_installed_sage_setup' into t/30277/remove_src_module_list_py`
`7f8ede4`	`Merge branch 't/30277/remove_src_module_list_py' into t/30163/cygwin__try_again_to_use_system_blas_and_system_r`

Answer 6 · 2020-08-14T00:14:23.000Z

Commit: 7f8ede4

Answer 7 · 2020-08-14T19:44:55.000Z

comment:4

cygwin-standard (https://github.com/mkoeppe/sage/runs/983346777?check_suite_focus=true) maplotlib build fails:

      File "/tmp/pip-req-build-ixb64a6o/setupext.py", line 461, in add_numpy_flags
        import numpy as np
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/__init__.py", line 143, in <module>
        from . import lib
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/lib/__init__.py", line 25, in <module>
        from .index_tricks import *
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/lib/index_tricks.py", line 11, in <module>
        import numpy.matrixlib as matrixlib
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/matrixlib/__init__.py", line 4, in <module>
        from .defmatrix import *
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/matrixlib/defmatrix.py", line 11, in <module>
        from numpy.linalg import matrix_power
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/linalg/__init__.py", line 73, in <module>
        from .linalg import *
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/linalg/linalg.py", line 33, in <module>
        from numpy.linalg import lapack_lite, _umath_linalg
    ImportError: No such file or directory

also scipy fails with the same error.

r system package is not used because prereq curl is not accepted:

Checking whether SageMath should install SPKG curl...
checking for curl 7.22... /usr/bin/curl
checking for gawk... (cached) gawk
checking for curl-config... /usr/bin/curl-config
checking for the version of libcurl... 7.71.1
checking for libcurl >= version 7.22... yes
checking whether libcurl is usable... no
checking whether programs linking to libcurl can be executed... no
configure: no suitable system package found for SPKG curl

and then building r also fails:

Warning in solve.default(rgb) :
  unable to load shared object '/cygdrive/d/a/sage/sage/local/var/tmp/sage/build/r-3.6.3/src/modules//lapack.dll':
  No such file or directory
Error in solve.default(rgb) : LAPACK routines cannot be loaded
Error: unable to load R code in package 'grDevices'
Execution halted
make[7]: *** [../../../share/make/lazycomp.mk:9: ../../../library/grDevices/R/grDevices.rdb] Error 1
make[7]: Target 'mklazycomp' not remade because of errors.
make[7]: Leaving directory '/cygdrive/d/a/sage/sage/local/var/tmp/sage/build/r-3.6.3/src/src/library/grDevices'
make[6]: *** [Makefile:28: all] Error 2
make[6]: Leaving directory '/cygdrive/d/a/sage/sage/local/var/tmp/sage/build/r-3.6.3/src/src/library/grDevices'
make[5]: *** [Makefile:39: R] Error 1
make[5]: Leaving directory '/cygdrive/d/a/sage/sage/local/var/tmp/sage/build/r-3.6.3/src/src/library'

Answer 8 · 2020-08-14T19:45:56.000Z

comment:5

Logs: https://github.com/mkoeppe/sage/suites/1048727679/artifacts/14102351

Answer 9 · 2020-08-14T19:50:39.000Z

comment:6

From the numpy build log:

blas_opt_info:
blas_mkl_info:
Disabled blas_mkl_info: (MKLROOT is None)
Disabled blas_mkl_info: (MKLROOT is None)
customize UnixCCompiler
  libraries mkl_rt not found in []
  NOT AVAILABLE

blis_info:
  libraries blis not found in ['/cygdrive/d/a/sage/sage/local/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

openblas_info:
Disabled openblas_info: (OPENBLAS is None)
  libraries openblas not found in []
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Disabled atlas_3_10_blas_threads_info: (PTATLAS is None)
  libraries tatlas not found in []
  NOT AVAILABLE

atlas_3_10_blas_info:
Disabled atlas_3_10_blas_info: (ATLAS is None)
  libraries satlas not found in []
  NOT AVAILABLE

atlas_blas_threads_info:
Disabled atlas_blas_threads_info: (PTATLAS is None)
  libraries ptf77blas,ptcblas,atlas not found in []
  NOT AVAILABLE

atlas_blas_info:
Disabled atlas_blas_info: (ATLAS is None)
  libraries f77blas,cblas,atlas not found in []
  NOT AVAILABLE

accelerate_info:
  NOT AVAILABLE

blas_info:
C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -fdebug-prefix-map=/cygdrive/d/cyg_pub/devel/python/python37/python37-3.7.7-1.x86_64/build=/usr/src/debug/python37-3.7.7-1 -fdebug-prefix-map=/cygdrive/d/cyg_pub/devel/python/python37/python37-3.7.7-1.x86_64/src/Python-3.7.7=/usr/src/debug/python37-3.7.7-1 -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -fdebug-prefix-map=/cygdrive/d/cyg_pub/devel/python/python37/python37-3.7.7-1.x86_64/build=/usr/src/debug/python37-3.7.7-1 -fdebug-prefix-map=/cygdrive/d/cyg_pub/devel/python/python37/python37-3.7.7-1.x86_64/src/Python-3.7.7=/usr/src/debug/python37-3.7.7-1

creating /tmp/tmp7np5_kr4/tmp
creating /tmp/tmp7np5_kr4/tmp/tmp7np5_kr4
compile options: '-I/usr/include -I/usr/include/suitesparse -I/cygdrive/d/a/sage/sage/local/include -c'
gcc: /tmp/tmp7np5_kr4/source.c
gcc /tmp/tmp7np5_kr4/tmp/tmp7np5_kr4/source.o -L/usr/lib -lblas -llapack -o /tmp/tmp7np5_kr4/a.out.exe
  FOUND:
    libraries = ['blas', 'lapack', 'blas']
    library_dirs = ['/usr/lib']
    include_dirs = ['/usr/include', '/usr/include/suitesparse', '/cygdrive/d/a/sage/sage/local/include']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

  FOUND:
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    libraries = ['blas', 'lapack', 'blas']
    library_dirs = ['/usr/lib']
    include_dirs = ['/usr/include', '/usr/include/suitesparse', '/cygdrive/d/a/sage/sage/local/include']
    language = c

lapack_opt_info:
lapack_mkl_info:
Disabled lapack_mkl_info: (MKLROOT is None)
Disabled lapack_mkl_info: (MKLROOT is None)
  libraries mkl_rt not found in []
  NOT AVAILABLE

openblas_lapack_info:
Disabled openblas_lapack_info: (OPENBLAS is None)
  libraries openblas not found in []
  NOT AVAILABLE

openblas_clapack_info:
Disabled openblas_clapack_info: (OPENBLAS is None)
  libraries openblas,lapack not found in []
  NOT AVAILABLE

flame_info:
  libraries flame not found in ['/cygdrive/d/a/sage/sage/local/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

atlas_3_10_threads_info:
Disabled atlas_3_10_threads_info: (PTATLAS is None)
<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
  NOT AVAILABLE

atlas_3_10_info:
Disabled atlas_3_10_info: (ATLAS is None)
<class 'numpy.distutils.system_info.atlas_3_10_info'>
  NOT AVAILABLE

atlas_threads_info:
Disabled atlas_threads_info: (PTATLAS is None)
<class 'numpy.distutils.system_info.atlas_threads_info'>
  NOT AVAILABLE

atlas_info:
Disabled atlas_info: (ATLAS is None)
<class 'numpy.distutils.system_info.atlas_info'>
  NOT AVAILABLE

lapack_info:
  FOUND:
    libraries = ['blas', 'lapack', 'lapack']
    library_dirs = ['/usr/lib']
    language = f77

  FOUND:
    libraries = ['blas', 'lapack', 'lapack', 'blas', 'lapack', 'blas']
    library_dirs = ['/usr/lib']
    language = c
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    include_dirs = ['/usr/include', '/usr/include/suitesparse', '/cygdrive/d/a/sage/sage/local/include']

.....





/cygdrive/d/a/sage/sage/local/var/tmp/sage/build/numpy-1.19.1/src/numpy/distutils/system_info.py:1914: UserWarning: 
    Optimized (vendor) Blas libraries are not found.
    Falls back to netlib Blas library which has worse performance.
    A better performance should be easily gained by switching
    Blas library.
  if self._calc_info(blas):

Answer 10 · 2020-08-14T20:04:01.000Z

Description changed:

--- 
+++ 
@@ -1 +1,5 @@
-... they were disabled in spkg-configure for 9.1 ...
+... they were disabled in spkg-configure for 9.1 in #29538 ...
+
+Previous system BLAS tickets:
+- #29025, #29051, #29398
+

Answer 11 · 2020-08-19T16:02:55.000Z

comment:9

Hi Erik, do you have time to look into this ticket?

Answer 12 · 2020-08-31T15:34:50.000Z

comment:10

This looks like a good start so far and your analysis of how to detect openblas makes sense. I'll look into it when I get back to my Windows machine.

Answer 13 · 2021-02-13T20:51:01.000Z

comment:11

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

Answer 14 · 2021-03-10T12:32:10.000Z

comment:12

It looks like one of the first things that needs to be fixed is properly detecting the system curl on Cygwin. I remember having trouble with this before. I thought I looked into it at one point, and I think even fixed it somehow, but now I don't remember for sure, or why the fix hasn't made it into Sage if indeed I did fix it...

Answer 15 · 2021-03-10T22:00:18.000Z

comment:13

No, libcurl is detected correctly, see https://github.com/mkoeppe/sage/runs/2070658581

Answer 16 · 2021-03-12T11:39:15.000Z

comment:14

Ok, great. I just mentioned it since one of your earlier comments on this thread said as much. But that was also several months ago. I could have sworn it was working now.