python3 spkg: Work around errors building/importing the _crypt module
mkoeppe opened this issue · 19 comments
Building the spkg python3 3.7.3.p1 fails on Debian with libc 2.28-10 when using the c-compiler package from conda-forge, as reported in #28745.
The _crypt extension module of Python is deprecated in 3.8 and will be removed in 3.10 according to https://www.python.org/dev/peps/pep-0594/
This ticket proposes to remove the module from our build of Python or to ignore errors found during the build.
CC: @isuruf @dimpase @jdemeyer
Component: packages: standard
Reviewer: Dima Pasechnik
Issue created by migration from https://trac.sagemath.org/ticket/29012
More information on the failure:
Python3's configure script correctly determines that crypt_r is available if linking -lcrypt.
But this information is not used when building the extension. Instead it links to crypt only if find_library_file can find libcrypt - which fails.
This flawed logic is still present in CPython master (3.9.0 alpha 2) at https://github.com/python/cpython/blob/master/setup.py (https://github.com/python/cpython/blob/7d6378051feeadf45b4ce45b4b406b65df255648/setup.py#L1035)
def detect_crypt(self):
# crypt module.
[...]
if self.compiler.find_library_file(self.lib_dirs, 'crypt'):
libs = ['crypt']
else:
libs = []
self.add(Extension('_crypt', ['_cryptmodule.c'],
libraries=libs))
}}
conda-forge has a patch for python3 that seems related, but does not fix this issue: https://github.com/conda-forge/python-feedstock/blob/3.8/recipe/patches/0012-Fix-find_library-so-that-it-looks-in-sys.prefix-lib-.patch
isn't it easy to patch cpython to avoid said find_library_file ?
E.g., make sure pyconfig.h.in gets a variable which is #defined if
r_crypt is available from libcrypt, and call sysconfig.get_config_var() in
setup.py. I did something like this in (rejected, for unrelated reasons) python/cpython#12825
we can do a cpython PR for this issue...
My guess is that this patch will be rejected because upstream is already phasing out crypt (and also nis, which is hitting a similar problem).
Update: It seems that installing the Debian package dpkg-dev (a dependency of build-essential), which provides dpkg-architecture, makes Python recognize the multiarch structure on Debian, and the problem seems to go away.
This is the code in https://github.com/python/cpython/blob/9f220e4968cf73fa60440120ee46881e7974e47d/setup.py#L548
What is the actual error from Python when building the crypt module? I've never had a problem with it before.
Replying to @embray:
What is the actual error from Python when building the crypt module? I've never had a problem with it before.
#28745 has the Dockerfile that can be used to reproduce it, based on continuumio/miniconda3:latest (Debian)
The errors after adding dpkg-dev look like this:
[python3-3.7.3.p1] LD_LIBRARY_PATH=/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src CC='/opt/conda/envs/sage/bin/x86_64-conda_cos6-linux-gnu-cc -pthread' LDSHARED='/opt/conda/envs/sage/bin/x86_64-
conda_cos6-linux-gnu-cc -pthread -shared -L/sage/local/lib -Wl,-rpath,/sage/local/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-
rpath,/opt/conda/envs/sage/lib -Wl,-rpath-link,/opt/conda/envs/sage/lib -L/opt/conda/envs/sage/lib -L. -L/sage/local/lib -Wl,-rpath,/sage/local/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,re
lro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/opt/conda/envs/sage/lib -Wl,-rpath-link,/opt/conda/envs/sage/lib -L/opt/conda/envs/sage/lib ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall'
_TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py build
[python3-3.7.3.p1] running build
[python3-3.7.3.p1] running build_ext
[python3-3.7.3.p1] INFO: Can't locate Tcl/Tk libs and/or headers
[python3-3.7.3.p1] building '_struct' extension
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules
[python3-3.7.3.p1] /opt/conda/envs/sage/bin/x86_64-conda_cos6-linux-gnu-cc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -march=nocona -mtune=haswell -ftr
ee-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/conda/envs/sage/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-i
nitializers -Werror=implicit-function-declaration -I./Include -I/sage/local/include -I. -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Incl
ude -I/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src -c /sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_struct.c -o build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3
-3.7.3.p1/src/Modules/_struct.o
[python3-3.7.3.p1] In file included from /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/features.h:356:0,
[python3-3.7.3.p1] from /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/limits.h:27,
[python3-3.7.3.p1] from /opt/conda/envs/sage/lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/include-fixed/limits.h:194,
[python3-3.7.3.p1] from /opt/conda/envs/sage/lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/include-fixed/syslimits.h:7,
[python3-3.7.3.p1] from /opt/conda/envs/sage/lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/include-fixed/limits.h:34,
[python3-3.7.3.p1] from ./Include/Python.h:11,
[python3-3.7.3.p1] from /sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_struct.c:8:
[python3-3.7.3.p1] /usr/include/x86_64-linux-gnu/sys/cdefs.h:482:49: error: missing binary operator before token "("
[python3-3.7.3.p1] #if __GNUC_PREREQ (4,8) || __glibc_clang_prereq (3,5)
[python3-3.7.3.p1] ^
[python3-3.7.3.p1] In file included from ./Include/Python.h:25:0,
[python3-3.7.3.p1] from /sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_struct.c:8:
[python3-3.7.3.p1] /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdio.h:49:1: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'typedef'
[python3-3.7.3.p1] typedef struct _IO_FILE FILE;
[python3-3.7.3.p1] ^~~~~~~
[python3-3.7.3.p1] /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdio.h:50:1: error: unknown type name '__END_NAMESPACE_STD'
[python3-3.7.3.p1] __END_NAMESPACE_STD
[python3-3.7.3.p1] ^~~~~~~~~~~~~~~~~~~
[python3-3.7.3.p1] /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdio.h: In function '__USING_NAMESPACE_STD':
[python3-3.7.3.p1] /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdio.h:65:25: error: storage class specified for parameter '__FILE'
followed by another 366MB of error messages
Yikes! I take it this is clang, not gcc?
This is mixing the sysroot in conda's gcc /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/ and the system headers in /usr/include/x86_64-linux-gnu/ which won't work.
Replying to @embray:
What is the actual error from Python when building the crypt module? I've never had a problem with it before.
See https://github.com/mkoeppe/sage/runs/412042067
for the logs
isuruf's comment on #29053:
Looks like python's setup.py doesn't find the conda compiler's sysroot. https://github.com/python/cpython/blob/v3.7.3/setup.py#L867
isuruf's comment on #29053: This patch might help https://github.com/conda-forge/python-feedstock/blob/master/recipe/patches/0014-Fix-cross-compilation-on-Debian-based-distros.patch
I don't think this patch can help -- we are not attempting a cross compile.
Moving tickets to milestone sage-9.2 based on a review of last modification date, branch status, and severity.
Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.
Outdated, should close
Reviewer: Dima Pasechnik