Conda build error
paskino opened this issue · 10 comments
paskino commented
After #183, on Jenkins the conda build fails with the following, seemingly not having made the package correctly?
The error seems to originate from CMake finding the base python of my mamba installation rather than the one used in the build environment
-- Found Python: /home/ofn77899/mambaforge/bin/python3.10 (found version "3.10.10") found components: Interpreter
======================================================================
ERROR: test_CPU_regularisers (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_CPU_regularisers
Traceback (most recent call last):
File "/home/jenkins/conda-bld/ccpi-regulariser_1710763552122/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/home/jenkins/conda-bld/ccpi-regulariser_1710763552122/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/home/jenkins/conda-bld/ccpi-regulariser_1710763552122/test_tmp/test/test_CPU_regularisers.py", line 6, in <module>
from ccpi.filters.regularisers import FGP_TV, SB_TV, TGV, LLT_ROF, FGP_dTV, NDF, Diff4th, ROF_TV, PD_TV
ModuleNotFoundError: No module named 'ccpi'
======================================================================
ERROR: test_run_test (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_run_test
Traceback (most recent call last):
File "/home/jenkins/conda-bld/ccpi-regulariser_1710763552122/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/home/jenkins/conda-bld/ccpi-regulariser_1710763552122/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/home/jenkins/conda-bld/ccpi-regulariser_1710763552122/test_tmp/test/test_run_test.py", line 5, in <module>
from ccpi.filters.regularisers import ROF_TV, FGP_TV, PD_TV, SB_TV, TGV, LLT_ROF, FGP_dTV, NDF, Diff4th
ModuleNotFoundError: No module named 'ccpi'
paskino commented
I tested on my machine and the conda build seems to work ???
paskino commented
I forced the latest conda on Jenkins and now the error is
AssertionError: Can't merge/copy source into subdirectory of itself. Please create separate spaces for these things.
src: /home/jenkins/workspace/CILsingle/CCPi-Regularisation-Toolkit
dst: /home/jenkins/workspace/CILsingle/CCPi-Regularisation-Toolkit/conda-bld/ccpi-regulariser_1710924290226/work
paskino commented
I ran the conda build locally and it failed on the following combination
_libgcc_mutex: 0.1-conda_forge conda-forge
_openmp_mutex: 4.5-2_gnu conda-forge
bzip2: 1.0.8-hd590300_5 conda-forge
ca-certificates: 2024.2.2-hbcca054_0 conda-forge
ccpi-regulariser: 3.0-np123py38_25 local
ld_impl_linux-64: 2.40-h41732ed_0 conda-forge
libblas: 3.9.0-21_linux64_openblas conda-forge
libcblas: 3.9.0-21_linux64_openblas conda-forge
libffi: 3.4.2-h7f98852_5 conda-forge
libgcc-ng: 13.2.0-h807b86a_5 conda-forge
libgfortran-ng: 13.2.0-h69a702a_5 conda-forge
libgfortran5: 13.2.0-ha4646dd_5 conda-forge
libgomp: 13.2.0-h807b86a_5 conda-forge
liblapack: 3.9.0-21_linux64_openblas conda-forge
libnsl: 2.0.1-hd590300_0 conda-forge
libopenblas: 0.3.26-pthreads_h413a1c8_0 conda-forge
libsqlite: 3.45.2-h2797004_0 conda-forge
libstdcxx-ng: 13.2.0-h7e041cc_5 conda-forge
libuuid: 2.38.1-h0b41bf4_0 conda-forge
libxcrypt: 4.4.36-hd590300_1 conda-forge
libzlib: 1.2.13-hd590300_5 conda-forge
ncurses: 6.4.20240210-h59595ed_0 conda-forge
numpy: 1.23.5-py38h7042d01_0 conda-forge
openssl: 3.2.1-hd590300_1 conda-forge
pip: 24.0-pyhd8ed1ab_0 conda-forge
python: 3.8.19-hd12c33a_0_cpython conda-forge
python_abi: 3.8-4_cp38 conda-forge
readline: 8.2-h8228510_1 conda-forge
setuptools: 69.2.0-pyhd8ed1ab_0 conda-forge
tk: 8.6.13-noxft_h4845f30_101 conda-forge
wheel: 0.43.0-pyhd8ed1ab_0 conda-forge
xz: 5.2.6-h166bdaf_0 conda-forge
paskino commented
@casperdcl on further investigation, conda build:
This is what the tar.bz2 produced by conda contains for python 3.9: that is it doesn't contain the python module
>>> tar tjf /home/ofn77899/mambaforge/conda-bld/broken/ccpi-regulariser-3.0-np124py39_26.tar.bz2
info/hash_input.json
info/files
info/test/run_test.sh
info/test/test/.vscode/settings.json
info/recipe/build.sh
info/index.json
info/paths.json
info/recipe/bld.bat
info/recipe/conda_build_config.yaml
info/recipe/meta.yaml.template
info/test/test/testroutines.py
info/recipe/meta.yaml
info/test/test/debug.py
info/about.json
info/test/test/test_CPU_regularisers.py
info/test/test/test_run_test.py
info/git
info/test/test/test_imageLena.bin
libcilreg.so
libcilregcuda.so
By comparison the same command produces the following for Python 3.10
tar tjf /home/ofn77899/mambaforge/conda-bld/linux-64/ccpi-regulariser-3.0-np124py310_25.tar.bz2
info/hash_input.json
info/test/run_test.sh
info/test/test/.vscode/settings.json
info/recipe/build.sh
info/index.json
info/recipe/bld.bat
info/recipe/conda_build_config.yaml
info/recipe/meta.yaml.template
info/test/test/testroutines.py
info/files
info/recipe/meta.yaml
info/test/test/debug.py
info/about.json
info/test/test/test_CPU_regularisers.py
info/paths.json
info/test/test/test_run_test.py
info/git
info/test/test/test_imageLena.bin
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/INSTALLER
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/METADATA
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/WHEEL
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/RECORD
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/REQUESTED
libcilreg.so
lib/python3.10/site-packages/ccpi/filters/libcilreg.so
libcilregcuda.so
lib/python3.10/site-packages/ccpi/filters/libcilregcuda.so
lib/python3.10/site-packages/ccpi/cuda_kernels/__init__.py
lib/python3.10/site-packages/ccpi/filters/utils.py
lib/python3.10/site-packages/ccpi/filters/regularisers.py
lib/python3.10/site-packages/ccpi/supp/qualitymetrics.py
lib/python3.10/site-packages/ccpi/filters/diffusion.py
lib/python3.10/site-packages/ccpi/filters/regularisers_old.py
lib/python3.10/site-packages/ccpi/filters/regularisersCuPy.py
lib/python3.10/site-packages/ccpi/filters/TV.py
lib/python3.10/site-packages/ccpi/supp/__init__.py
lib/python3.10/site-packages/ccpi/filters/__init__.py
lib/python3.10/site-packages/ccpi/__init__.py
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/top_level.txt
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/direct_url.json
lib/python3.10/site-packages/ccpi/__pycache__/__init__.cpython-310.pyc
lib/python3.10/site-packages/ccpi/supp/__pycache__/__init__.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/__init__.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/utils.cpython-310.pyc
lib/python3.10/site-packages/ccpi/cuda_kernels/__pycache__/__init__.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/regularisers.cpython-310.pyc
lib/python3.10/site-packages/ccpi/supp/__pycache__/qualitymetrics.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/diffusion.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/regularisers_old.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/regularisersCuPy.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/TV.cpython-310.pyc
paskino commented
I think that was the python
about this, what was the working (3.10) package list?
That was the python 3.10 that I showed above. Does this answer you?
I checked that my system created the following conda packages before failing on the 3.9
ccpi-regulariser-3.0-np121py310_25.tar.bz2
ccpi-regulariser-3.0-np121py310_35.tar.bz2
ccpi-regulariser-3.0-np124py310_25.tar.bz2
ccpi-regulariser-3.0-np122py310_25.tar.bz2
ccpi-regulariser-3.0-np123py310_25.tar.bz2
ccpi-regulariser-3.0-np124py311_25.tar.bz2
ccpi-regulariser-3.0-np123py311_25.tar.bz2
paskino commented
I don't understand what you are asking.
I ran conda build recipe -c conda-forge --python=3.8, that produced a conda package which didn't contain the python code:
>>> tar tjf /home/ofn77899/mambaforge/conda-bld/broken/ccpi-regulariser-3.0-np123py38_25.tar.bz2
info/hash_input.json
info/files
info/test/run_test.sh
info/test/test/.vscode/settings.json
info/recipe/build.sh
info/index.json
info/paths.json
info/recipe/bld.bat
info/recipe/conda_build_config.yaml
info/recipe/meta.yaml.template
info/test/test/testroutines.py
info/recipe/meta.yaml
info/test/test/debug.py
info/about.json
info/test/test/test_CPU_regularisers.py
info/test/test/test_run_test.py
info/git
info/test/test/test_imageLena.bin
libcilreg.so
libcilregcuda.so
The same happens with python 3.8 and 3.9, not on 3.10 and 3.11.
What package could have changed between 3.9 and 3.10 that produces this error? We are using pyproject.toml, which is new to me.
paskino commented
Ah, I understand now! This is the environment that was created with py3.10 is the following
The following NEW packages will be INSTALLED:
_libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
_openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
bzip2 conda-forge/linux-64::bzip2-1.0.8-hd590300_5
ca-certificates conda-forge/linux-64::ca-certificates-2024.2.2-hbcca054_0
ccpi-regulariser home/ofn77899/mambaforge/conda-bld/linux-64::ccpi-regulariser-3.0-np121py310_35
ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.40-h41732ed_0
libblas conda-forge/linux-64::libblas-3.9.0-21_linux64_openblas
libcblas conda-forge/linux-64::libcblas-3.9.0-21_linux64_openblas
libffi conda-forge/linux-64::libffi-3.4.2-h7f98852_5
libgcc-ng conda-forge/linux-64::libgcc-ng-13.2.0-h807b86a_5
libgfortran-ng conda-forge/linux-64::libgfortran-ng-13.2.0-h69a702a_5
libgfortran5 conda-forge/linux-64::libgfortran5-13.2.0-ha4646dd_5
libgomp conda-forge/linux-64::libgomp-13.2.0-h807b86a_5
liblapack conda-forge/linux-64::liblapack-3.9.0-21_linux64_openblas
libnsl conda-forge/linux-64::libnsl-2.0.1-hd590300_0
libopenblas conda-forge/linux-64::libopenblas-0.3.26-pthreads_h413a1c8_0
libsqlite conda-forge/linux-64::libsqlite-3.45.2-h2797004_0
libstdcxx-ng conda-forge/linux-64::libstdcxx-ng-13.2.0-h7e041cc_5
libuuid conda-forge/linux-64::libuuid-2.38.1-h0b41bf4_0
libxcrypt conda-forge/linux-64::libxcrypt-4.4.36-hd590300_1
libzlib conda-forge/linux-64::libzlib-1.2.13-hd590300_5
ncurses conda-forge/linux-64::ncurses-6.4.20240210-h59595ed_0
numpy conda-forge/linux-64::numpy-1.21.6-py310h45f3432_0
openssl conda-forge/linux-64::openssl-3.2.1-hd590300_1
pip conda-forge/noarch::pip-24.0-pyhd8ed1ab_0
python conda-forge/linux-64::python-3.10.14-hd12c33a_0_cpython
python_abi conda-forge/linux-64::python_abi-3.10-4_cp310
readline conda-forge/linux-64::readline-8.2-h8228510_1
setuptools conda-forge/noarch::setuptools-69.2.0-pyhd8ed1ab_0
tk conda-forge/linux-64::tk-8.6.13-noxft_h4845f30_101
tzdata conda-forge/noarch::tzdata-2024a-h0c530f3_0
wheel conda-forge/noarch::wheel-0.43.0-pyhd8ed1ab_0
xz conda-forge/linux-64::xz-5.2.6-h166bdaf_0
The unittest ran correctly.