possible regression? cx_freeze 6.16 - 7.1 for possibly scipy
jontis opened this issue · 20 comments
We have a scientific app based on QT and the scipy stack, compiling with cx_freeze for linux and windows targets.
It worked fine with dev version cx_freeze 6.16 that was available during 2023 december.
trying to compile now with another system with cx_freeze 7.1, it fails.
We have compared the environments and believe that the only difference is versi5on of cx_freeze but can absolutely have missed something.
We don't dare to mess with the only functioning build system we have now.
The main packages are:
python 3.10
scikit-learn 1.3.2 py310h1fdf081_1 conda-forge
numpy 1.26.0 py310hb13e2d6_0 conda-forge
pandas 2.1.3 py310hcc13569_0 conda-forge
pyqt 5.15.9 py310h04931ad_5 conda-forge
pyqt5-sip 12.12.2 py310hc6cd4ac_5 conda-forge
qt-main 5.15.8 h01ceb2d_12 conda-forge
We build with:
python setup.py bdist_appimage > cxlog.txt
setup.py:
import toml # for reading pyproject.toml
pyproject = toml.load("../pyproject.toml")
version = pyproject['project']['version']
print(f"sys.platform: {sys.platform}")
base = "Win32GUI" if sys.platform == "win32" else None
script_path = "main.py"
include_files = ["lib/", ("../pyproject.toml", "lib/pyproject.toml"),]
exe = Executable(
script=script_path,
)
# Setup cx_Freeze options.
options = {
"build_exe": {
"includes": [],
"excludes": [],
"packages": ["pyabf", "igor2", "tqdm", "sklearn", "scipy", "seaborn"],
"include_files": include_files
}
}
# Call the setup function.
setup(
name="brainwash",
version=version,
description="",
#packages=find_packages(where="src"),
#package_dir={"": "src"},
#include_package_data=True,
options=options,
executables=[exe],
base=base
)
The error when trying to run appimage built by cx_freeze 7.1 is:
$ dist/brainwash-0.9.0-x86_64.AppImage
/tmp/.mount_brainwHL6FMz/main: error while loading shared libraries: libcrypt.so.2: cannot open shared object file: No such file or directory
Please test with cx_Freeze 7.1.1
I have tested some samples of pandas, scikit-learn, using a newer version, and they have worked.
For instance, I have tested, using scikit-learn 1.5.0 and an example from sklearn auto_examples_python/bicluster, used bdist_appimage and it works.
Tested with cx_freeze 7.1.1 from conda. Same problem. It says very early in the process:
sys.platform: linux
running bdist_appimage
/home/jonathan/.local/bin/appimagetool --version
running build_exe
patchelf --version returns: 'patchelf 0.17.2\n'
WARNING: cannot find 'libcrypt.so.2'
I've checked and while libcrypt.so.2 is present in several other env, it is not in this env.
Missing dependancy for packaging?
I solved that error by:
mamba install libxcrypt
this provided libcrypt.so.2
Now on to next error.
File "/home/jonathan/mambaforge/envs/brainwash/lib/python3.10/site-packages/scipy/linalg/blas.py", line 213, in
from scipy.linalg import _fblas
ImportError: libcblas.so.3: cannot open shared object file: No such file or directory
This file is already present in env:
$ locate libcblas.so.3
/home/jonathan/mambaforge/envs/brainwash/lib/libcblas.so.3
And the only mention of it in the build log is:
$ cat cxlog.txt | grep libcblas
patchelf --replace-needed libcblas.so.3 libopenblasp-r0.3.24.so /home/jonathan/code/brainwash/src/build/exe.linux-x86_64-3.10/lib/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so returns: ''
Try to upgrade numpy
cx_Freeze have a sample: https://github.com/marcelotduarte/cx_Freeze/tree/main/samples/pandas
I tested it with numpy 1.26.4 and pandas 2.2.2 and works.
You can use it to test your environment.
Tried but no change. It may be scipy that is the problem, and it seems to work for cx_freeze 6.16
Error message:
File "/home/jonathan/mambaforge/envs/brainwash/lib/python3.10/site-packages/scipy/linalg/blas.py", line 213, in
from scipy.linalg import _fblas
ImportError: libcblas.so.3: cannot open shared object file: No such file or directory
Triggered by:
from scipy.signal import savgol_filter, find_peaks
Tried also bumping scipy to latest 1.13.1 but the error remains unchanged.
Checked for differences in blas in both functioning cxfreeze 6.16 and non functioning cxfreeze 7.1.1 system but they are the same:
libblas 3.9.0 19_linux64_openblas conda-forge
libcblas 3.9.0 19_linux64_openblas conda-forge
liblapack 3.9.0 19_linux64_openblas conda-forge
libopenblas 0.3.24 pthreads_h413a1c8_0 conda-forge
Few questions:
- The pandas sample works in your side?
- How you have installed numpy?
- Can you test using:
3.1) conda install "libblas=*=mkl" numpy
or
3.2) conda install "libblas==*openblas" numpy - You can indicate a sample/example of code using scipy to test on my side?
Tried also bumping scipy to latest 1.13.1 but the error remains unchanged.
In your first comment of this thread, you do not listed the scipy version.
Checking the sources, the lasted tested version is 1.11.2. Can you test with it?
I'll try find a example to test the latest version.
Ok, results of the tests:
4. to test, I just added to the top of your pandas sample test_pandas.py: from scipy import signal
1, your pandas sample works both in its minimal env and our production env.
3. linux comes with openblas as default, and we're running that for both linux and windows as the mkl is huge with very little performance gains.
3.1 you line doesn't work to switch to mkl, but this did: conda install "libblas=*=*openblas" numpy. The pandas sample with scipy signal builds but does not run. It crashes with:
Traceback (most recent call last):
File "/home/jonathan/mambaforge/envs/cxtest/lib/python3.12/site-packages/cx_Freeze/initscripts/startup.py", line 141, in run
module_init.run(name + "main")
File "/home/jonathan/mambaforge/envs/cxtest/lib/python3.12/site-packages/cx_Freeze/initscripts/console.py", line 25, in run
exec(code, main_globals)
File "test_pandas.py", line 5, in
from scipy import signal
...
...
from scipy.linalg import _fblas
ImportError: libcblas.so.3: cannot open shared object file: No such file or directory
3.2 builds with openblas but crashes the same way as did with mkl
3.1 for me is not working too w/ the pandas sample.
In the docs for scipy signal: https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.correlate.html#scipy.signal.correlate I get a example to test
scipy 1.11.2 and scipy 1.13.1 fails
I'll work on this.
We tried to compile this example with the cxfreeze 6.16 dev and it seems to work fine.
I did some tests and noticed that the regression was caused by #2419, but resolved in #2466 when adding support for numpy 2.0.
I also tested the current versions in conda, and verified that 7.0 (conda install cx_freeze=7.0) works with scipy 1.11.3 up to the most recent 1.14.0, as long as it uses numpy 1.26.4 (may be other minor versions).
Versions 7.1 and 7.1.1 are broken for use with scipy, but work with numpy, pandas, etc.
I tried to backport the 2466 but apparently, it didn't work, but if you can, try it (7.1.1-2 or cx_freeze-7.1.1-py312h41a817b_2.conda).
But installing via pip worked, with all mentioned versions of scipy and also with numpy 2.0. To test:
conda uninstall cx_freeze
pip uninstall cx_freeze
pip install --no-deps --no-build-isolation git+https://github.com/marcelotduarte/cx_Freeze.git@refs/pull/2466/head
or
pip install --no-deps --no-build-isolation git+https://github.com/marcelotduarte/cx_Freeze.git@develop
Greetings!
Your pandas sample (and our own project) now builds and runs on popOS, but Miniforge3 on Win10 seems to have a problem:
If I install cx_Freeze by "pip install --no-deps --no-build-isolation git+https://github.com/marcelotduarte/cx_Freeze.git@refs/pull/2466/head",
then typing "python setup.py build_exe" returns the following error:
Traceback (most recent call last):
File "C:\Users\matso\cx_Freeze\samples\pandas\setup.py", line 18, in
executables = [Executable("test_pandas.py")]
File "C:\Users\matso\miniforge3\envs\cx_freeze_pandas\lib\site-packages\cx_Freeze\executable.py", line 47, in init
self.base = base
File "C:\Users\matso\miniforge3\envs\cx_freeze_pandas\lib\site-packages\cx_Freeze\executable.py", line 87, in base
raise OptionError(msg)
cx_Freeze.exception.OptionError: no base named 'console' ('console-cpython-310-win_amd64')
If I just install by "pip install cx_Freeze", it builds, but running the .exe returns
Traceback (most recent call last):
File "C:\Users\matso\miniforge3\envs\cx_freeze_pandas\Lib\site-packages\cx_Freeze\initscripts_startup_.py", line 141, in run
module_init.run(name + "main")
File "C:\Users\matso\miniforge3\envs\cx_freeze_pandas\Lib\site-packages\cx_Freeze\initscripts\console.py", line 25, in run
exec(code, main_globals)
File "test_pandas.py", line 4, in
File "C:\Users\matso\miniforge3\envs\cx_freeze_pandas\lib\site-packages\pandas_init_.py", line 59, in
from pandas.core.api import (
File "C:\Users\matso\miniforge3\envs\cx_freeze_pandas\lib\site-packages\pandas\core\api.py", line 1, in
from pandas.libs import (
File "C:\Users\matso\miniforge3\envs\cx_freeze_pandas\lib\site-packages\pandas_libs_init.py", line 18, in
from pandas._libs.interval import Interval
File "interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
This happens whether I try to add scipy.signal to the test or not.
Version list:
python 3.10.13, h4de0772_1_cpython, conda-forge
pip 24.0, pyhd8ed1ab_0, conda-forge
cx-freeze 7.1.1, pypi_0
numpy 2.0.0, pypi_0
pandas 2.1.4, pypi_0
scipy 1.14.0, pypi_0
Let me know if you need more information!
// Mats
You can test the latest development build:
pip install --force --no-cache --pre --extra-index-url https://marcelotduarte.github.io/packages/ cx_Freeze
It seems the problems still arise when trying to import scipy in windows builds.
I've upgraded to python 3.12.4, pandas 2.2.2, cx-freeze 7.2.0.dev44.
Building and running still works in popOS, and it works for your pandas sample in windows, now!
However, attempting to import scipy (1.14.0) in the same example yields the following traceback:
Traceback (most recent call last):
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\cx_Freeze\initscripts\__startup__.py", line 141, in run
module_init.run(name + "__main__")
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\cx_Freeze\initscripts\console.py", line 25, in run
exec(code, main_globals)
File "test_pandas.py", line 5, in <module>
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\scipy\signal\__init__.py", line 307, in <module>
from . import _sigtools, windows
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\scipy\signal\windows\__init__.py", line 42, in <module>
from ._windows import *
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\scipy\signal\windows\_windows.py", line 7, in <module>
from scipy import linalg, special, fft as sp_fft
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\scipy\__init__.py", line 147, in __getattr__
return _importlib.import_module(f'scipy.{name}')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\matso\miniforge3\envs\cx\Lib\importlib\__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\scipy\linalg\__init__.py", line 205, in <module>
from ._basic import *
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\scipy\linalg\_basic.py", line 13, in <module>
from ._decomp import _asarray_validated
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\scipy\linalg\_decomp.py", line 24, in <module>
from scipy._lib._util import _asarray_validated
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\scipy\_lib\_util.py", line 18, in <module>
from scipy._lib._array_api import array_namespace, is_numpy, size as xp_size
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\scipy\_lib\_array_api.py", line 21, in <module>
from scipy._lib.array_api_compat import (
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\scipy\_lib\array_api_compat\numpy\__init__.py", line 1, in <module>
from numpy import * # noqa: F403
^^^^^^^^^^^^^^^^^^^
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\numpy\__init__.py", line 366, in __getattr__
import numpy.testing as testing
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\numpy\testing\__init__.py", line 8, in <module>
from unittest import TestCase
ModuleNotFoundError: No module named 'unittest'
If you use a the same setup as panda sample, in this line: https://github.com/marcelotduarte/cx_Freeze/blob/main/samples/pandas/setup.py#L22
remove the "unittest" to see if it works.
That got me a "ModuleNotFoundError: No module named 'pydoc'." - so I set the exclude list to [], and then it ran on Windows, too. Thank you so much!
However, now that scipy runs, our project seems to stumble on scikit-learn.
If I add the line "from sklearn import linear_model" to test_pandas.py, it builds, but running it I get the following traceback:
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\cx_Freeze\initscripts_startup_.py", line 141, in run
module_init.run(name + "main")
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\cx_Freeze\initscripts\console.py", line 25, in run
exec(code, main_globals)
File "test_pandas.py", line 6, in
File "C:\Users\matso\miniforge3\envs\cx\Lib\site-packages\sklearn_init_.py", line 80, in
from . import (
File "C:/Users/matso/miniforge3/envs/cx/Lib/site-packages/sklearn/distributor_init.py", line 21, in
WinDLL(op.abspath(vcomp140_dll_filename))
File "C:\Users\matso\miniforge3\envs\cx\Lib\ctypes_init.py", line 379, in init
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: Could not find module 'C:\Users\matso\cx_Freeze\samples\pandas\build\exe.win-amd64-3.12\vcomp140.dll' (or one of its dependencies). Try using the full path with constructor syntax.
I've added the following lines to setup.py:
vcomp140_dll_path = os.path.join(os.environ['windir'], 'System32', 'vcomp140.dll')
print(f"vcomp140_dll_path: {vcomp140_dll_path}, exists: {os.path.exists(vcomp140_dll_path)}")
options = {
"build_exe": {
"excludes": [],
"include_files": [f"{vcomp140_dll_path}"],
},
}
The print confirms that the .dll exists when building, but the error persists (there is no vcomp140.dll in the build directory, where the program looks for it).
Release 7.2.0 is out!
Documentation