Python 3.12 support

Question

Python 3.12 support

edumorlom opened this issue a year ago · 36 comments

edumorlom commented a year ago

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
Support for python 3.12 which is officially out.

Answer 1 · 2023-10-06T04:48:15.000Z

Internal: Epic https://jira.rocketsoftware.com/browse/DBC-14681 opened for this enhancement request. Thanks.

Answer 2 · 2024-01-08T07:37:56.000Z

Can you provide any insight when we can expect a release supporting 3.12?

Answer 3 · 2024-01-08T07:42:43.000Z

@LarsMichelsen We have started working on support for python 3.12.
The Internal: Epic https://jira.rocketsoftware.com/browse/DBC-14681 is in progress now.
Will try to finish as soon as possible.

Thank you for your patience
Balram

Answer 4 · 2024-01-11T21:02:53.000Z

@LarsMichelsen We have started working on support for python 3.12.

Great :)

Will try to finish as soon as possible.

When reading this, shall we think of days, weeks or months? Don't want to be pushy, just asking for better understanding.

Answer 5 · 2024-01-12T04:12:51.000Z

@LarsMichelsen We have started working on support for python 3.12.

Great :)

Will try to finish as soon as possible.

When reading this, shall we think of days, weeks or months? Don't want to be pushy, just asking for better understanding.

It may take some weeks, but will try to deliver as soon as possible.

Answer 6 · 2024-01-18T17:00:26.000Z

Installed Python 3.12.0 on z/OS 2.5. Attached is the output from the pip install ibm-db --verbose

python312_ibm-db.txt

Answer 7 · 2024-01-19T03:39:11.000Z

@gngrossi
Yes, the error is because of 'distutils' module, the support for this module is removed from python 3.12.x

Answer 8 · 2024-01-19T08:29:26.000Z

@bchoudhary6415 any workaround for this other than running on 3.11?

Answer 9 · 2024-01-19T09:11:40.000Z

@bchoudhary6415 any workaround for this other than running on 3.11?

@yadukrishnanperumpillil No, as legacy Unicode C APIs have been removed from python 3.12 onwards,
Please refer the link - https://docs.python.org/3.12/whatsnew/3.12.html#id10
We have already created the Internal: Epic https://jira.rocketsoftware.com/browse/DBC-14681 to support for python 3.12.
Epic is in progress.

Thank you

Answer 10 · 2024-01-19T14:50:30.000Z

@gngrossi Current release of ibm_db i.e. v3.2.1 do not support Python 3.12. Hence, error is expected.
Next release of ibm_db should support 3.12, till then you can use Python 3.11. Thanks.

Answer 11 · 2024-01-19T15:21:28.000Z

Will the new release of ibm-db be able to be compiled with IBM C/C++ for Open Enterprise Languages on z/OS 2.0.0?
Since there is a need to let the pip installer know where the Python include/library directories are,
is there an environment variable like XL_CONFIG that can be specified? (used it for xlclang.cfg).

Would one of these work?
CLANG_CONFIG_FILE_USER_DIR
CLANG_CONFIG_FILE_SYSTEM_DIR

clang.cfg

#-fzos-le-char-mode=ascii
#-mzos-hlq-csslib=SYS1 -mzos-hlq-clang=SYS1

-Wall -m64 -mrent -march=z15 -mzos-hlq-le=SYS1 -O2 -I/hewitt/zopentools/python-3.12/usr/lpp/IBM/cyp/v3r12/pyz/include/python3.12
-L/hewitt/zopentools/python-3.12/usr/lpp/IBM/cyp/v3r12/pyz/lib

thanks

Answer 12 · 2024-01-31T05:16:32.000Z

@bimalkjha
Is there any tentative date or ETA?
Not pushing, just curious so we can plan it out.

Answer 13 · 2024-02-09T14:46:34.000Z

Thanks for your effort. I have the same issue here using 3.12 :(

Answer 14 · 2024-02-15T14:43:51.000Z

PR #913 opened to deliver fix for this issue. Thanks.

Answer 15 · 2024-02-19T08:44:58.000Z

@bimalkjha Thank you very much for expediting this.
I see PR is merged now, when do we expect the release artifact out.

Answer 16 · 2024-02-19T09:32:44.000Z

Next week, by EOM. Thanks.

Answer 17 · 2024-02-29T22:36:18.000Z

Will the new release of ibm-db be able to be compiled with IBM C/C++ for Open Enterprise Languages on z/OS 2.0.0?
Since there is a need to let the pip installer know where the Python include/library directories are,
is there an environment variable like XL_CONFIG that can be specified? (used it for xlclang.cfg).

When will the package be available on pypi.org?

thanks

Answer 18 · 2024-03-01T06:28:55.000Z

Hello @gngrossi

You can create ". profile" file and specify the python path environment variables such as
export PATH=$HOME/bin:/user/python_install/bin:$PATH
export LIBPATH=$HOME/lib:/user/python_install/lib:$PATH

For more info regarding ibm_db Installation on z/OS please refer the link - https://github.com/ibmdb/python-ibmdb/blob/master/INSTALL.md#inszos

Thank you

Answer 19 · 2024-03-02T14:33:48.000Z

@bimalkjha i see that the python3.12 support was merged to main branch but it has not been released to pypi yet.

https://pypi.org/project/ibm-db/

Answer 20 · 2024-03-02T18:27:41.000Z

@edumorlom Just released. Please install and verify. Thanks.

Answer 21 · 2024-03-02T21:14:32.000Z

Do I include this in my install script?
thanks

export SETUPTOOLS_USE_DISTUTILS=stdlib

Answer 22 · 2024-03-04T08:17:07.000Z

Hello @gngrossi
No, need to add "export SETUPTOOLS_USE_DISTUTILS=stdlib" in your install script as from python 3.12 onwards distutils package is removed - please refer the link - https://docs.python.org/3/whatsnew/3.12.html#distutils
the setuptools project can be installed: it still provides distutils.

Thanks

Answer 23 · 2024-03-04T15:32:26.000Z

@bchoudhary6415 Thanks, I will remove the export SETUPTOOLS_USE_DISTUTILS=stdlib from the script.

If I recall, I needed it for Python 3.11.5 but the doc says otherwise.
https://docs.python.org/3/whatsnew/3.12.html#distutils

Answer 24 · 2024-03-05T04:28:16.000Z

@gngrossi
Actually distutils was in deprecation from Python 3.10,
And from Python 3.12 onwards it's removed.
That is why, we used setuptools in Python 3.12 support.

Thanks

Answer 25 · 2024-03-05T22:38:24.000Z

Getting an error during the link. I need to use the xlclang.cfg so qarch=13,-qoptimize=2,-I and -L can be used.
I pointed to it using export XL_CONFIG="/hewitt/zopentools/installs/xlclang.cfg" and ran the install from the directory.

python312_ibmdb322.txt

xlclang.cfg.txt

Answer 26 · 2024-03-06T06:33:53.000Z

@gngrossi
Are you able to install ibm_db without setting XL_CONFIG environment variable?

Answer 27 · 2024-03-06T23:12:27.000Z

@bchoudhary6415 No, receiving the same error. The -L directory doesn't exist.

xlclang build/temp.os390-28.00-3931-cpython-312/ibm_db.o /u/@02858/tmp/pip-install-dg_gnhbu/ibm-db_a0b5efaf392342a0883dd48243edfd68/libdsnao64c.x -L. -L/usr/lpp/IBM/cyp/v3r12/pyz/lib -o build/lib.os390-28.00-3931-cpython-312/ibm_db.cpython-312.so
FSUM3248 xlclang: Directory /usr/lpp/IBM/cyp/v3r12/pyz/lib specified by command option -L cannot be searched.

python312_ibmdb322_withoutXL_CONFIG.txt

Answer 28 · 2024-03-07T04:26:01.000Z

@gngrossi
I can see from the log file you shared the error you got is
ERROR: Could not build wheels for ibm-db, which is required to install pyproject.toml-based projects

in case there is error in install due to pyproject.toml file try the following approach

To install the package ibm_db it is necessary at first to install the build dependency package - wheel:
pip3 install wheel
Install ibm_db with the pip flag --no-build-isolation:
pip3 install ibm_db --no-build-isolation

Answer 29 · 2024-03-07T19:58:51.000Z

@bchoudhary6415
Is installing wheel required Python 3.12 and ibm-db? For Python 3.11.x, it wasn't.

Prior to trying ibm-db with Python 3.12, I was able to install these packages including upgrading pip.
pip3 install boto3
pip3 install Flask
pip3 install pymongo
pip3 install redis
pip3 install requests
pip3 install zoautil_py-1.3.0.3-cp312-none-any.whl

bash-5.2$ pip3 list
Package Version

blinker 1.7.0
boto3 1.34.22
botocore 1.34.22
certifi 2023.11.17
cffi 1.14.6
charset-normalizer 3.3.2
click 8.1.7
cryptography 3.3.2
dnspython 2.4.2
Flask 3.0.1
idna 3.6
itsdangerous 2.1.2
Jinja2 3.1.3
jmespath 1.0.1
MarkupSafe 2.1.3
pip 24.0
pycparser 2.20
pymongo 4.6.1
python-dateutil 2.8.2
redis 5.0.1
requests 2.31.0
s3transfer 0.10.0
six 1.16.0
urllib3 2.0.7
Werkzeug 3.0.1
zoautil-py 1.3.0.3

I still need to be able to use the xlclang.cfg file with the install.
thanks

Answer 30 · 2024-03-08T04:25:01.000Z

@gngrossi
With the above steps, ibm_db is installed or not?
Please refer this issue - #707 for more info
With lower python version such as 3.11.x
Are you able to install ibm_db with export XL_CONFIG="/hewitt/zopentools/installs/xlclang.cfg ?
I tried creating xlclang.cfg file and setting XL_CONFIG variable to that file and I'm able to install ibm_db
Please check the attached text file
xlclang.cfg.txt
Also before installing ibm_db refer this link https://github.com/ibmdb/python-ibmdb/blob/master/INSTALL.md#inszos and follow the steps

Thanks

Answer 31 · 2024-03-11T22:00:39.000Z

@bchoudhary6415 For 3.12, without using the XL_CONFIG, ibm-db was not installed.
With 3.11, I was able to install-db using export XL_CONFIG="$HOME/c/xlclang.cfg" and specifying the -I and -L directories for python 311.

Answer 32 · 2024-03-11T22:14:48.000Z

@bchoudhary6415
Attached are logs from running after installing wheel 0.42.0

pip3 install ibm-db --no-build-isolation --verbose
python312_ibm-db_no-build-isolation_3-11-2024.txt

pip3 install ibm-db --verbose
python312_ibm-db_3-11-2024.txt

I referenced the install.md and I'm not seeing where I use an xlclang.cfg file. With the cfg file, besides supplying the Python include and lib directories, I'd like to optimize for running on a z15.

thanks

Answer 33 · 2024-03-12T05:46:47.000Z

@gngrossi
I see the below error
xlclang build/temp.os390-28.00-3931-cpython-312/ibm_db.o /u/@02858/tmp/pip-install-42t3f0qd/ibm-db_c5da6aa13f9344e0a79c2620290f15fe/libdsnao64c.x -L. -L/usr/lpp/IBM/cyp/v3r12/pyz/lib -o build/lib.os390-28.00-3931-cpython-312/ibm_db.cpython-312.so
FSUM3248 xlclang: Directory /usr/lpp/IBM/cyp/v3r12/pyz/lib specified by command option -L cannot be searched.
error: command '/bin/xlclang' failed with exit code 40
error: subprocess-exited-with-error

Here the error is fsum3248 - please refer the link - https://www.ibm.com/docs/en/zos/3.1.0?topic=fsum-fsum3248
Please ensure that the Directory can be searched.
Can you check that directory path is correct and have permission?

Thanks

Answer 34 · 2024-03-12T23:03:58.000Z

I installed Python from the pax file into /hewitt/zopentools/python312/usr/lpp/IBM/cyp/v3r12/pyz

The xlclang.cfg contains the include and lib directories along with -qarch=13,-qoptimize=2
-I/hewitt/zopentools/python-3.12/usr/lpp/IBM/cyp/v3r12/pyz/include/python3.12,-L/hewitt/zopentools/python-3.12/usr/lpp/IBM/cyp/v3r12/pyz/lib

Here is my Python sourcing profile.

profile.python

export PYZ_HOME="/hewitt/zopentools/python312/usr/lpp/IBM/cyp/v3r12/pyz"
export PATH=${PYZ_HOME}/bin:$PATH
export LIBPATH=${PYZ_HOME}/lib:$LIBPATH
export _BPXK_AUTOCVT=ON
export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
export _TAG_REDIR_IN=txt
export _TAG_REDIR_OUT=txt
export _TAG_REDIR_ERR=txt

Recommended Configuration for IBM z/OS XL C/C++

export _CC_CCMODE=1
export _CXX_CCMODE=1
export _C89_CCMODE=1
export _CC_EXTRA_ARGS=1
export _CXX_EXTRA_ARGS=1
export _C89_EXTRA_ARGS=1

Build packages with setuptools

export CC=xlclang
export CXX=xlclang
export LDSHARED=xlclang

Answer 35 · 2024-03-14T13:19:24.000Z

Here's the portion of the xlclang.cfg file specifying options for -I and -L with directories that should be used.
The compile step looks to be using my -I specified directory, how?
Others options from the xlclang.cfg file don't appear as being used (-qarch=13,-qoptimize=2).
The link step is missing my -L directory as it's defaulting to IBM's which doesn't exist.

Should specifying the environment variable for the location of xlclang.cfg allow /bin/xlclang to use the -F parameter?

Clang C compiler
clang: use = DEFLT
Clang C++ compiler
clang++: use = DEFLT
options = -D_XOPEN_SOURCE=600,-D__static_assert=static_assert,-Wno-parentheses,-Wno-unused-value
common definitions
DEFLT: cppcomp = /usr/lpp/cbclib/xlclang/exe/clcdrvr
ccomp = /usr/lpp/cbclib/xlclang/exe/clcdrvr
as = /bin/c89
ld_c = /bin/c89
ld_cpp = /bin/cxx
xlC = /usr/lpp/cbclib/xlclang/bin/xlclang
xlCcopt = -D_XOPEN_SOURCE
sysobj = sys1.sceeobj:sys1.sceecpp
syslib_x = sys1.sceebnd2:sys1.csslib
exportlist_c_x = sys1.sceelib(celhs003,celhs001)
exportlist_cpp_x = sys1.sceelib(celhs003,celhs001,celhscpp)
exportlist_c_64 = sys1.sceelib(celqs003)
exportlist_cpp_64 = sys1.sceelib(celqs003,celqscpp,cxxrt64)
cinc = -isystem/usr/include/le
cppinc = -isystem/usr/include/c++
options = -D_UNIX03_WITHDRAWN,-L/usr/lpp/cbclib/lib,-qarch=13,-qoptimize=2,-I/hewitt/zopentools/python-3.12/usr/lpp/IBM/cyp/v3r12/pyz/include/python3.12,-L/hewitt/zopentools/python-3.12/usr/lpp/IBM/cyp/v3r12/pyz/lib
libraries = -libmcmp
steplib = sys1.sclccmp

--

/bin/xlclang -DNDEBUG -O3 -D_XOPEN_SOURCE_EXTENDED -D_UNIX03_THREADS -D_POSIX_THREADS -D_OPEN_SYS_FILE_EXT -I/u/@02858/tmp/pip-install-a_0v9ss5/ibm-db_54993ad519164773833cf75400197a34/sdsnc.h -I/hewitt/zopentools/python312/usr/lpp/IBM/cyp/v3r12/pyz/include/python3.12 -c ibm_db.c -o build/temp.os390-28.00-3931-cpython-312/ibm_db.o

/bin/xlclang build/temp.os390-28.00-3931-cpython-312/ibm_db.o /u/@02858/tmp/pip-install-a_0v9ss5/ibm-db_54993ad519164773833cf75400197a34/libdsnao64c.x -L. -L/usr/lpp/IBM/cyp/v3r12/pyz/lib -o build/lib.os390-28.00-3931-cpython-312/ibm_db.cpython-312.so
FSUM3248 xlclang: Directory /usr/lpp/IBM/cyp/v3r12/pyz/lib specified by command option -L cannot be searched.
error: command '/bin/xlclang' failed with exit code 40

Answer 36 · 2024-04-01T20:06:13.000Z

workaround until resolved

https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/steven-pitman1/2023/11/17/installing-pypi-packages-with-c-components-on-pyth

Would one of these work? CLANG_CONFIG_FILE_USER_DIR CLANG_CONFIG_FILE_SYSTEM_DIR

profile.python

Recommended Configuration for IBM z/OS XL C/C++

Build packages with setuptools

workaround until resolved

Would one of these work?
CLANG_CONFIG_FILE_USER_DIR
CLANG_CONFIG_FILE_SYSTEM_DIR