Python 3.9 Build Error
Opened this issue · 15 comments
Building against CPython 3.9 I see the following errors:
/home/conda/feedstock_root/build_artifacts/adios-python_1602593380049/_build_env/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it with " \
^~~~~~~
adios.cpp: In function 'PyObject* PyInit_adios()':
adios.cpp:52250:30: error: 'PyTypeObject {aka struct _typeobject}' has no member named 'tp_print'; did you mean 'tp_dict'?
__pyx_type_5adios_softdict.tp_print = 0;
^~~~~~~~
tp_dict
Refs.: https://github.com/conda-forge/adios-python-feedstock
cc @jychoi-hpc
Ping @jychoi-hpc @pnorbert @dmitry-ganyushin @lwan86: do you think this can be easily fixed? :)
@ax3l sorry for my late reply. Can you help me to understand the error? Is this because of using new version of numpy? which python version did you use? I tried to build on Cori with python 3.9.7 (numpy 1.20.3, cython 0.29.24) and didn't get an error.
@jychoi-hpc no worries. I think this is due to a new version of CPython. tp_print is dropped from Python 3.9 onwards.
https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_vectorcall_offset
Maybe Cython of some of the checked-in ADIOS Cython calls/file generations need to be updated?
I see the problem on conda-forge:
https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_vectorcall_offset
and build like this:
https://github.com/conda-forge/adios-python-feedstock/blob/master/recipe/build.sh
Should I run Cython or something beforehand? :)
I just tried again by doing exactly the same thing in build.sh on Cori without any issue. I used Python 3.9.7. It looks like condo-forge is using more recent python. Do you know which version of python on conda-forge is used?
Alternatively, you can try the following command to invoke 'cython' beforehand:
make CYTHON=y python
instead of
make python
Thanks a lot, let's try this in Spack (spack/spack#28003 / spack/spack#28223) and Conda-Forge (conda-forge/adios-python-feedstock#11).
X-ref: #202
Works, fantastic! Thanks for the hint!
I noticed that the makefile does not reliably re-create the .cpp files from the .pyx files. In Spack, I had to remove them beforehand and disable make parallelism (-j) to get it to build: spack/spack#28223
Looks like we still hit this error with suggested commands:
[ 214s] + make CYTHON=y MPI=y python
[ 214s] python setup.py build_ext -lrt
[ 214s] running build_ext
[ 214s] building 'adios.adios' extension
[ 214s] creating build
[ 214s] creating build/temp.linux-aarch64-3.9
[ 214s] gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -D_GNU_SOURCE -fPIC -fwrapv -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -I/opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include -I/home/abuild/rpmbuild/BUILD/adios-1.13.1/src/public -L/home/abuild/rpmbuild/BUILD/adios-1.13.1/src -I/opt/ohpc/pub/mpi/openmpi4-gnu12/4.1.4/include -fPIC -D_NOMPI -I/opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/adios/1.13.1/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/netcdf/4.9.0/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/hdf5/1.10.8/include -I/usr/include -I/usr/include/python3.9 -c adios.cpp -o build/temp.linux-aarch64-3.9/adios.o -Wno-uninitialized -Wno-unused-function
[ 214s] In file included from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
[ 214s] from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
[ 214s] from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
[ 214s] from adios.cpp:540:
[ 214s] /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
[ 214s] 17 | #warning "Using deprecated NumPy API, disable it with " \
[ 214s] | ^~~~~~~
[ 215s] adios.cpp: In function ‘PyObject* PyInit_adios()’:
[ 215s] adios.cpp:52250:30: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52250 | __pyx_type_5adios_softdict.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52261:26: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52261 | __pyx_type_5adios_file.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52287:31: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52287 | __pyx_type_5adios_blockinfo.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52300:25: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52300 | __pyx_type_5adios_var.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52306:26: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52306 | __pyx_type_5adios_attr.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52312:27: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52312 | __pyx_type_5adios_group.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52338:31: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52338 | __pyx_type_5adios_smartdict.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52343:28: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52343 | __pyx_type_5adios_writer.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52348:30: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52348 | __pyx_type_5adios_attrinfo.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52353:29: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52353 | __pyx_type_5adios_varinfo.tp_print = 0;
[ 215s] | ^~~~~~~~
Any suggestion?
Trying to build ADIOS 1.13.1 (latest release, from Apr 18 2018) fails with the above problem.
adios.cpp and adios_mpi.cpp are regenerated with Cython 0.29 at Oct 2 2019 with 327e159
I think it is time for a new release!
It looks like the cython version 0.29 at Oct 2 2019 is old. I just tried adios 1.13.1, cython 0.29.32, and python 3.9.13 and it works for me:
jyc@login4 numpy ((v1.13.1)) $ cython --version
Cython version 0.29.32
jyc@login4 numpy ((v1.13.1)) $ python --version
Python 3.9.13
jyc@login4 numpy ((v1.13.1)) $ make CYTHON=y MPI=y python
cython --cplus adios_mpi.pyx
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /gpfs/alpine/csc143/world-shared/jyc/summit/src/adios_develop/wrappers/numpy/adios_mpi.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
cython --cplus adios.pyx
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /gpfs/alpine/csc143/world-shared/jyc/summit/src/adios_develop/wrappers/numpy/adios.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
python setup.py build_ext -lrt
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
running build_ext
building 'adios.adios' extension
gcc -pthread -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -mcpu=power8 -fPIC -O2 -isystem /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -mcpu=power8 -fPIC -O2 -isystem /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -fPIC -D_NOMPI -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include/python3.9 -c adios.cpp -o build/temp.linux-ppc64le-cpython-39/adios.o -Wno-uninitialized -Wno-unused-function
In file included from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from adios.cpp:761:
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with " \
| ^~~~~~~
g++ -pthread -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -shared -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib build/temp.linux-ppc64le-cpython-39/adios.o -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/lib -ladios_nompi -lrt -lm -lrt -o build/lib.linux-ppc64le-cpython-39/adios/adios.cpython-39-powerpc64le-linux-gnu.so
python setup_mpi.py build
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
running build
running build_src
running build_py
copying src_mpi/__init__.py -> build/lib.linux-ppc64le-cpython-39/adios_mpi
running build_ext
MPI configuration: [mpi] from 'mpi.cfg'
MPI C compiler: /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicc
MPI C++ compiler: /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicxx
MPI F77 compiler: /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpif77
MPI F90 compiler: /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpif90
building 'adios_mpi.adios_mpi' extension
/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicc -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include/python3.9 -c adios_mpi.cpp -o build/temp.linux-ppc64le-cpython-39/adios_mpi.o -Wno-unused-function -Wno-uninitialized
In file included from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from adios_mpi.cpp:762:
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with " \
| ^~~~~~~
/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicxx -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -shared -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib build/temp.linux-ppc64le-cpython-39/adios_mpi.o -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/lib -ladios -lrt -lm -o build/lib.linux-ppc64le-cpython-39/adios_mpi/adios_mpi.cpython-39-powerpc64le-linux-gnu.so
(removed some warnings)
I don't know which version of cython starts supproting python 3.9. But, can you try with different cython versions?
I wonder how this works for you with v1.13.1!
https://raw.githubusercontent.com/ornladios/ADIOS/v1.13.1/wrappers/numpy/adios.cpp does not use checks like #if PY_VERSION_HEX < 0x030800B1 before trying to access .tp_print as it does in master
I'll try to investigate some more tomorrow!
@martin-g @Yikun you are probably aware (and don't let me break our enthusiasm on getting old builds to work), but ADIOS1 is now deprecated and ADIOS2 is the new hot project: https://github.com/ornladios/ADIOS2
I used the following command to re-generate adios.cpp and adios_mpi.cpp:
make CYTHON=y MPI=y python
As @ax3l said, I also recommend to use Adios2 if possible.
Thank you for your help, @jychoi-hpc and @ax3l !
I didn't know it is possible to re-generate these files on the fly!
This is what we did at https://github.com/openhpc/ohpc/pull/1635/files
If we face other issues most probably we will move to ADIOS 2!
If we face other issues most probably we will move to ADIOS 2!
ADIOS2 also has more features that have been maintained for a few years.