undefined symbol: Py_SET_TYPE
grisuthedragon opened this issue · 6 comments
After installing slycot using pip I got the following error, when I am trying to import it:
Python 3.8.10 (default, Nov 14 2022, 12:59:47)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import slycot
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/user/env/lib/python3.8/site-packages/slycot/__init__.py", line 16, in <module>
from .analysis import ab01nd, ab05md, ab05nd, ab07nd, ab08nd, ab08nz
File "/home/user/env/lib/python3.8/site-packages/slycot/analysis.py", line 22, in <module>
from . import _wrapper
ImportError: /home/user/env/lib/python3.8/site-packages/slycot/_wrapper.cpython-38-x86_64-linux-gnu.so: undefined symbol: Py_SET_TYPEI am using Ubuntu 20.04 with Python 3.8.10 and created a new virtual environment for it:
$ virtualenv env
$ source ./env/bin/activate
$ pip install --no-cache-dir slycot
Collecting slycot
Downloading slycot-0.5.3.tar.gz (3.0 MB)
|████████████████████████████████| 3.0 MB 6.3 MB/s
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Collecting numpy
Downloading numpy-1.24.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
|████████████████████████████████| 17.3 MB 100.9 MB/s
Building wheels for collected packages: slycot
Building wheel for slycot (PEP 517) ... done
Created wheel for slycot: filename=slycot-0.5.3-cp38-cp38-linux_x86_64.whl size=1987476 sha256=53f161973476b615f3a07211877910eb89ad85d66428fc31d8b6e980a21a766a
Stored in directory: /tmp/pip-ephem-wheel-cache-e2j324rb/wheels/f7/b1/ef/08bc53cdb0fe98fa3c4067f1e674ebc4fd32a965544b558450
Successfully built slycot
Installing collected packages: numpy, slycot
Successfully installed numpy-1.24.2 slycot-0.5.3
$ python -c 'import slycot'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/user/env/lib/python3.8/site-packages/slycot/__init__.py", line 16, in <module>
from .analysis import ab01nd, ab05md, ab05nd, ab07nd, ab08nd, ab08nz
File "/home/user/env/lib/python3.8/site-packages/slycot/analysis.py", line 22, in <module>
from . import _wrapper
ImportError: /home/user/env/lib/python3.8/site-packages/slycot/_wrapper.cpython-38-x86_64-linux-gnu.so: undefined symbol: Py_SET_TYPEHi @grisuthedragon, thanks for the report.
The error message is interesting and would imply that the compilation step during pip install would have found the wrong python headers. Could you run and post the output of a verbose compilation please?
pip wheel -v slycot
I can reproduce this (also Ubuntu 20.04).
Output of (rm -rf env/ && virtualenv env && source ./env/bin/activate && pip wheel -v slycot) |& tee slycot-build-v.txt attached.
This looks fine to me:
/tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/cmake/data/bin/cmake /tmp/pip-wheel-rr0bha7n/slycot -G Ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-wheel-rr0bha7n/slycot/_skbuild/linux-x86_64-3.8/cmake-install -DPYTHON_VERSION_STRING:STRING=3.8.10 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/home/rory/tmp/env/bin/python -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPYTHON_LIBRARY:PATH=/usr/lib/x86_64-linux-gnu/libpython3.8.so -DPython_EXECUTABLE:PATH=/home/rory/tmp/env/bin/python -DPython_ROOT_DIR:PATH=/home/rory/tmp/env -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPython_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/numpy/core/include -DPython3_EXECUTABLE:PATH=/home/rory/tmp/env/bin/python -DPython3_ROOT_DIR:PATH=/home/rory/tmp/env -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPython3_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/numpy/core/include -DCMAKE_BUILD_TYPE:STRING=Release
specifically
-DPython_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/numpy/core/include
and
-DPython3_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/numpy/core/include
followed by CMake reporting:
-- Found NumPy: /tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/numpy/core/include (found version "1.24.3")
-- Found F2PY: /tmp/pip-build-env-_qcqcl1f/overlay/bin/f2py3 (found version "1.24.3")
This compiler warning, however, suggests wrong Numpy (and Python?) headers are used (I have numpy installed in base system via apt)
[684/1236] Building C object slycot/CMakeFiles/_wrapper.dir/_wrappermodule.c.o
/tmp/pip-wheel-rr0bha7n/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c: In function ‘int_from_pyobj’:
/tmp/pip-wheel-rr0bha7n/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c:271:14: warning: implicit declaration of function ‘Npy__PyLong_AsInt’ [-Wimplicit-function-declaration]
271 | *v = Npy__PyLong_AsInt(obj);
| ^~~~~~~~~~~~~~~~~
/tmp/pip-wheel-rr0bha7n/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c: In function ‘PyInit__wrapper’:
/tmp/pip-wheel-rr0bha7n/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c:26712:5: warning: implicit declaration of function ‘Py_SET_TYPE’; did you mean ‘Py_TYPE’? [-Wimplicit-function-declaration]
26712 | Py_SET_TYPE(&PyFortran_Type, &PyType_Type);
| ^~~~~~~~~~~
| Py_TYPE
I'm going to uninstall system numpy and python3-dev, and try again.
My comment about Python headers doesn't make sense: we're using the system Python in a venv, so we must use system headers in some way. Having said that I did uninstall those apt packages (and python3.8-dev), and I could successfully build. This makes sense i.t.o. numpy (see also below), but not, I think, python3-dev. I'm a bit confused about this.
I reinstalled python3-dev and numpy via apt, and did a build from git working tree branch from v0.5.4. The command python setup.py build -- -- -v gives this:
[680/1236] /usr/bin/cc -D_wrapper_EXPORTS -I/usr/include/python3.8 -I/home/rory/projects/slycot/gh193-buildbug/venv/lib/python3.8/site-packages/numpy/core/include -I/home/rory/projects/slycot/gh193-buildbug/venv/lib/python3.8/site-packages/numpy/f2py/src -O3 -DNDEBUG -fPIC -MD -MT slycot/CMakeFiles/_wrapper.dir/_wrappermodule.c.o -MF slycot/CMakeFiles/_wrapper.dir/_wrappermodule.c.o.d -o slycot/CMakeFiles/_wrapper.dir/_wrappermodule.c.o -c /home/rory/src/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c
/home/rory/src/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c: In function ‘int_from_pyobj’:
/home/rory/src/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c:271:14: warning: implicit declaration of function ‘Npy__PyLong_AsInt’ [-Wimplicit-function-declaration]
271 | *v = Npy__PyLong_AsInt(obj);
| ^~~~~~~~~~~~~~~~~
/home/rory/src/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c: In function ‘PyInit__wrapper’:
/home/rory/src/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c:26712:5: warning: implicit declaration of function ‘Py_SET_TYPE’; did you mean ‘Py_TYPE’? [-Wimplicit-function-declaration]
26712 | Py_SET_TYPE(&PyFortran_Type, &PyType_Type);
| ^~~~~~~~~~~
| Py_TYPE
If I look in /usr/include/python3.8:
$ ls -l /usr/include/python3.8|grep numpy
lrwxrwxrwx 1 root root 56 Dec 5 2022 numpy -> ../../lib/python3/dist-packages/numpy/core/include/numpy
My guess is gcc's include ordering gives preference to the system Numpy, since -I/usr/include/python3.8 is first in the arguments relative to the venv Numpy dirs. I think we rely on scikit-build to make that command-line, I don't know if we can affect the order of output.
So, I don't know how to fix this, but a workaround is to uninstall system numpy (apt remove python3-numpy), do the venv setup, then reinstall system numpy.
FWIW, the Ubuntu package for development is libpython3.8-dev; removing this prevents a successful build.
Nice digging @roryyorke. I am very surprised that this is an issue. We are certainly not the only package compiling stuff with both numpy and python headers, using the scikit-build cmake chain. What do we do different to other packages?