sagemath/sage

ubuntu-bionic: python 3.8.5 readline module does not build after installing ncurses, readline spkgs

guenterrote opened this issue · 20 comments

guenterrote commented

running on Ubuntu 18.04. I get the same result on the "master" and the current "develop" branch

config-log and build-log is attached.

The bug seems to have something to do with these error messages:

*** WARNING: renaming "readline" since importing it failed: /home/rote/Dokumente/Software/sage/local/lib/libreadline.so.8: undefined symbol: UP


[python3-3.8.5] Following modules built successfully but were removed because they could not be imported:
[python3-3.8.5] readline

This error can be reproduced using:

tox -e docker-ubuntu-bionic-standard -- ncurses readline python3

Depends on #32137

CC: @dimpase @zlscherr @kliem

Component: build

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/31267

guenterrote commented

python build logfile

guenterrote commented

Attachment: python3-3.8.5.log

config.log

guenterrote commented

Description changed:

--- 
+++ 
@@ -2,7 +2,10 @@
 
 config-log and build-log is attached.
 
-The bug seems to have something to do with this error message:
+The bug seems to have something to do with these error messages:
+
+*** WARNING: renaming "readline" since importing it failed: /home/rote/Dokumente/Software/sage/local/lib/libreadline.so.8: undefined symbol: UP
+
 
 [python3-3.8.5] Following modules built successfully but were removed because they could not be imported:
 [python3-3.8.5] readline
guenterrote commented
comment:1

Attachment: config.log

guenterrote commented

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-running on Ubuntu 18.04
+running on Ubuntu 18.04. I get the same result on the "master" and the current "develop" branch
 
 config-log and build-log is attached.
mkoeppe commented
comment:3

From python3...log:

checking for getc_unlocked() and friends... yes
checking how to link readline libs... -lreadline -ltinfo
...

but when python3 actually links readline, it is doing something else:

building 'readline' extension
gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -O2 -g -march=native -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I./Include/internal -I./Include -I/home/rote/Dokumente/Software/sage/local/include -I. -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/home/rote/Dokumente/Software/sage/local/var/tmp/sage/build/python3-3.8.5/src/Include -I/home/rote/Dokumente/Software/sage/local/var/tmp/sage/build/python3-3.8.5/src -c /home/rote/Dokumente/Software/sage/local/var/tmp/sage/build/python3-3.8.5/src/Modules/readline.c -o build/temp.linux-x86_64-3.8/home/rote/Dokumente/Software/sage/local/var/tmp/sage/build/python3-3.8.5/src/Modules/readline.o
gcc -pthread -shared -Wl,-rpath-link,/home/rote/Dokumente/Software/sage/local/lib -L/home/rote/Dokumente/Software/sage/local/lib -Wl,-rpath,/home/rote/Dokumente/Software/sage/local/lib -L. -Wl,-rpath-link,/home/rote/Dokumente/Software/sage/local/lib -L/home/rote/Dokumente/Software/sage/local/lib -Wl,-rpath,/home/rote/Dokumente/Software/sage/local/lib -L. -Wl,-rpath-link,/home/rote/Dokumente/Software/sage/local/lib -L/home/rote/Dokumente/Software/sage/local/lib -Wl,-rpath,/home/rote/Dokumente/Software/sage/local/lib build/temp.linux-x86_64-3.8/home/rote/Dokumente/Software/sage/local/var/tmp/sage/build/python3-3.8.5/src/Modules/readline.o -L/usr/lib/termcap -L. -L/home/rote/Dokumente/Software/sage/local/lib -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -lreadline -lncursesw -o build/lib.linux-x86_64-3.8/readline.cpython-38-x86_64-linux-gnu.so

You could check if uninstalling libncursesw5-dev solves this problem

mkoeppe commented
comment:4

https://github.com/python/cpython/blob/master/configure.ac#L4887 checks it in the order "" tinfo ncursesw ncurses curses termcap

https://github.com/python/cpython/blob/master/setup.py#L975 detect_readline_curses redoes the checks in a different way

mkoeppe commented
comment:5

Günter, on a typical linux machine, one would actually expect that ncurses and readline can be used and do not have to be built from scratch. So I would recommend to do make ncurses-clean readline-clean and then re-run configure. This should be more robust than the current configuration

mkoeppe commented

Description changed:

--- 
+++ 
@@ -9,3 +9,13 @@
 
 [python3-3.8.5] Following modules built successfully but were removed because they could not be imported:
 [python3-3.8.5] readline   
+
+
+This error can be reproduced using:
+
+```
+tox -e docker-ubuntu-bionic-standard -- ncurses readline python3
+```
+
+
+
mkoeppe commented

Description changed:

--- 
+++ 
@@ -4,12 +4,13 @@
 
 The bug seems to have something to do with these error messages:
 
+```
 *** WARNING: renaming "readline" since importing it failed: /home/rote/Dokumente/Software/sage/local/lib/libreadline.so.8: undefined symbol: UP
 
 
 [python3-3.8.5] Following modules built successfully but were removed because they could not be imported:
 [python3-3.8.5] readline   
-
+```
 
 This error can be reproduced using:
mkoeppe commented
comment:8
root@4d2478da7517:/sage# ldd local/lib/libreadline.so.8
	linux-vdso.so.1 (0x00007ffc7a3c8000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fee30793000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fee30dd5000)
root@4d2478da7517:/sage# ldd /usr/lib/x86_64-linux-gnu/libreadline.so 
	linux-vdso.so.1 (0x00007ffe23ded000)
	libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f100ed8d000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f100e99c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f100f200000)

The issue likely arises because find_library_file inspects the system library but then the Sage-built library is used for linking https://github.com/python/cpython/blob/master/setup.py#L977

I think this is a limitation of distutils - it does not know about LIBRARY_PATH

mkoeppe commented
comment:9

I think this would need a similar fix as it needed in the packaging of python3 in homebrew...

guenterrote commented
comment:11

Replying to @mkoeppe:

Günter, on a typical linux machine, one would actually expect that ncurses and readline can be used and do not have to be built from scratch. So I would recommend to do make ncurses-clean readline-clean and then re-run configure. This should be more robust than the current configuration

I tried this; the python installation seemed to go through fine, but eventually the build failed.

cd ../.. && sage-logger -p './sage --docbuild --no-pdf-links all html ' logs/dochtml.log
[dochtml] Traceback (most recent call last):
...
[dochtml]   File "sage/matrix/matrix_mpolynomial_dense.pyx", line 1, in init sage.matrix.matrix_mpolynomial_dense (build/cythonized/sage/matrix/matrix_mpolynomial_dense.cpp:8672)
[dochtml] ImportError: libreadline.so.8: cannot open shared object file: No such file or directory

Trying to start sage anyway displays a nice banner but crashes with the same error

---------------------------------------------------------------------------
ImportErrorPython 3.8.5: /home/rote/Dokumente/Software/sage/local/bin/python3
                                                   Thu Jan 21 07:42:55 2021
A problem occurred executing Python code.  Here is the sequence of function
...
...
~/Dokumente/Software/sage/local/lib/python3.8/site-packages/sage/matrix/matrix_mpolynomial_dense.pyx in init sage.matrix.matrix_mpolynomial_dense (build/cythonized/sage/matrix/matrix_mpolynomial_dense.cpp:8672)()
----> 1 """
      2 Dense matrices over multivariate polynomials over fields
...
ImportError: libreadline.so.8: cannot open shared object file: No such file or directory
mkoeppe commented
comment:12

Replying to @guenterrote:

Replying to @mkoeppe:

Günter, on a typical linux machine, one would actually expect that ncurses and readline can be used and do not have to be built from scratch. So I would recommend to do make ncurses-clean readline-clean and then re-run configure. This should be more robust than the current configuration

I tried this; the python installation seemed to go through fine, but eventually the build failed.

cd ../.. && sage-logger -p './sage --docbuild --no-pdf-links all html ' logs/dochtml.log
[dochtml] Traceback (most recent call last):
...
[dochtml]   File "sage/matrix/matrix_mpolynomial_dense.pyx", line 1, in init sage.matrix.matrix_mpolynomial_dense (build/cythonized/sage/matrix/matrix_mpolynomial_dense.cpp:8672)
[dochtml] ImportError: libreadline.so.8: cannot open shared object file: No such file or directory

Trying to start sage anyway displays a nice banner but crashes with the same error

---------------------------------------------------------------------------
ImportErrorPython 3.8.5: /home/rote/Dokumente/Software/sage/local/bin/python3
                                                   Thu Jan 21 07:42:55 2021
A problem occurred executing Python code.  Here is the sequence of function
...
...
~/Dokumente/Software/sage/local/lib/python3.8/site-packages/sage/matrix/matrix_mpolynomial_dense.pyx in init sage.matrix.matrix_mpolynomial_dense (build/cythonized/sage/matrix/matrix_mpolynomial_dense.cpp:8672)()
----> 1 """
      2 Dense matrices over multivariate polynomials over fields
...
ImportError: libreadline.so.8: cannot open shared object file: No such file or directory

This looks like make singular-clean sagelib-clean build could fix it

guenterrote commented
comment:13

Replying to @mkoeppe:

This looks like make singular-clean sagelib-clean build could fix it

This worked! Thanks!

mkoeppe commented
comment:14

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

mkoeppe commented
comment:18

Probably because of our ncurses/readline update in #32137, tox -e docker-ubuntu-bionic-standard -- ncurses readline python3 now works.

mkoeppe commented

Dependencies: #32137

dimpase commented

Reviewer: Dima Pasechnik

dimpase commented
comment:19

ok