CMake fails on pi with bookworm release
biol75 opened this issue · 3 comments
An up-to-dated pi, with bookworm, and git clone dies with
CMake Error at blosc2/CMakeLists.txt:25 (include_directories):
include_directories given empty-string as include directory.
full output.....
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
Cloning into 'python-blosc2'...
remote: Enumerating objects: 4954, done.
remote: Counting objects: 100% (4954/4954), done.
remote: Compressing objects: 100% (1379/1379), done.
remote: Total 4954 (delta 3325), reused 4912 (delta 3307), pack-reused 0
Receiving objects: 100% (4954/4954), 6.19 MiB | 1.76 MiB/s, done.
Resolving deltas: 100% (3325/3325), done.
Submodule 'blosc2/c-blosc2' (https://github.com/Blosc/c-blosc2.git) registered for path 'blosc2/c-blosc2'
Cloning into '/home/pi/python-blosc2/blosc2/c-blosc2'...
Submodule path 'blosc2/c-blosc2': checked out 'f8417b103e6b0bbe06b861f92d57285590e1166a'
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: setuptools in /home/pi/venv/lib/python3.9/site-packages (from -r requirements-build.txt (line 1)) (68.2.2)
Collecting scikit-build
Using cached https://www.piwheels.org/simple/scikit-build/scikit_build-0.17.6-py3-none-any.whl (84 kB)
Collecting cython
Using cached https://www.piwheels.org/simple/cython/Cython-3.0.4-cp39-cp39-linux_armv7l.whl (10.7 MB)
Collecting cmake
Using cached https://www.piwheels.org/simple/cmake/cmake-3.27.7-cp39-cp39-linux_armv7l.whl (19.3 MB)
Collecting oldest-supported-numpy
Using cached https://www.piwheels.org/simple/oldest-supported-numpy/oldest_supported_numpy-2023.10.25-py3-none-any.whl (4.9 kB)
Collecting ninja
Using cached https://www.piwheels.org/simple/ninja/ninja-1.11.1.1-cp39-cp39-linux_armv7l.whl (132 kB)
Collecting build
Downloading https://www.piwheels.org/simple/build/build-1.0.3-py3-none-any.whl (18 kB)
Requirement already satisfied: packaging>=19.0 in /home/pi/venv/lib/python3.9/site-packages (from build->-r requirements-build.txt (line 7)) (23.2)
Requirement already satisfied: tomli>=1.1.0 in /home/pi/venv/lib/python3.9/site-packages (from build->-r requirements-build.txt (line 7)) (2.0.1)
Collecting importlib-metadata>=4.6
Downloading https://www.piwheels.org/simple/importlib-metadata/importlib_metadata-6.8.0-py3-none-any.whl (22 kB)
Collecting pyproject_hooks
Downloading https://www.piwheels.org/simple/pyproject-hooks/pyproject_hooks-1.0.0-py3-none-any.whl (9.3 kB)
Collecting zipp>=0.5
Using cached https://www.piwheels.org/simple/zipp/zipp-3.17.0-py3-none-any.whl (7.4 kB)
Collecting numpy==1.19.3
Using cached https://www.piwheels.org/simple/numpy/numpy-1.19.3-cp39-cp39-linux_armv7l.whl (10.5 MB)
Collecting wheel>=0.32.0
Using cached https://www.piwheels.org/simple/wheel/wheel-0.41.2-py3-none-any.whl (64 kB)
Collecting distro
Using cached https://www.piwheels.org/simple/distro/distro-1.8.0-py3-none-any.whl (20 kB)
Installing collected packages: zipp, wheel, pyproject-hooks, numpy, importlib-metadata, distro, scikit-build, oldest-supported-numpy, ninja, cython, cmake, build
Successfully installed build-1.0.3 cmake-3.27.7 cython-3.0.4 distro-1.8.0 importlib-metadata-6.8.0 ninja-1.11.1.1 numpy-1.19.3 oldest-supported-numpy-2023.10.25 pyproject-hooks-1.0.0 scikit-build-0.17.6 wheel-0.41.2 zipp-3.17.0
-- Trying 'Ninja' generator
--
Not searching for unused variables given on the command line.
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument value or use a ... suffix to tell
CMake that the project does not need compatibility with older versions.
-- The C compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- The CXX compiler identification is GNU 10.2.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (1.7s)
-- Generating done (0.0s)
-- Build files have been written to: /home/pi/python-blosc2/_cmake_test_compile/build
-- Trying 'Ninja' generator - success
Configuring Project
Working directory:
/home/pi/python-blosc2/_skbuild/linux-armv7l-3.9/cmake-build
Command:
/home/pi/venv/lib/python3.9/site-packages/cmake/data/bin/cmake /home/pi/python-blosc2 -G Ninja -DCMAKE_MAKE_PROGRAM:FILEPATH=/home/pi/venv/lib/python3.9/site-packages/ninja/data/bin/ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/home/pi/python-blosc2/_skbuild/linux-armv7l-3.9/cmake-install -DPYTHON_VERSION_STRING:STRING=3.9.2 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/home/pi/venv/lib/python3.9/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/home/pi/venv/bin/python -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.9 -DPYTHON_LIBRARY:PATH=/usr/lib/arm-linux-gnueabihf/libpython3.9.so -DPython_EXECUTABLE:PATH=/home/pi/venv/bin/python -DPython_ROOT_DIR:PATH=/home/pi/venv -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.9 -DPython3_EXECUTABLE:PATH=/home/pi/venv/bin/python -DPython3_ROOT_DIR:PATH=/home/pi/venv -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.9 -DCMAKE_MAKE_PROGRAM:FILEPATH=/home/pi/venv/lib/python3.9/site-packages/ninja/data/bin/ninja -DCMAKE_BUILD_TYPE:STRING=Release
Not searching for unused variables given on the command line.
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Warning (dev) at /home/pi/venv/lib/python3.9/site-packages/skbuild/resources/cmake/FindPythonExtensions.cmake:245 (find_package):
Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
are removed. Run "cmake --help-policy CMP0148" for policy details. Use
the cmake_policy command to set the policy and suppress this warning.
Call Stack (most recent call first):
CMakeLists.txt:3 (find_package)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Found PythonInterp: /home/pi/venv/bin/python (found version "3.9.2")
CMake Warning (dev) at /home/pi/venv/lib/python3.9/site-packages/skbuild/resources/cmake/FindPythonExtensions.cmake:252 (find_package):
Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
are removed. Run "cmake --help-policy CMP0148" for policy details. Use
the cmake_policy command to set the policy and suppress this warning.
Call Stack (most recent call first):
CMakeLists.txt:3 (find_package)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Found PythonLibs: /usr/lib/arm-linux-gnueabihf/libpython3.9.so (found version "3.9.2")
-- Found Cython: /home/pi/venv/bin/cython
-- Found NumPy: /usr/include/python3.9 (found version "")
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
Configuring for Blosc version: 2.10.5
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Using LZ4 internal sources.
-- Using ZLIB-NG internal sources for ZLIB support.
-- Using CMake version 3.27.7
-- ZLIB_HEADER_VERSION: 1.2.11
-- ZLIBNG_HEADER_VERSION: 2.0.7
-- Arch detected: 'armv6'
-- Basearch of 'armv6' has been detected as: 'arm'
-- Performing Test HAVE_FLOATABI_SOFTFP
-- Performing Test HAVE_FLOATABI_SOFTFP - Failed
-- Performing Test HAVE_FLOATABI_HARD
-- Performing Test HAVE_FLOATABI_HARD - Success
-- ARM floating point arch: -mfloat-abi=hard
-- Performing Test MFPU_NEON_AVAILABLE
-- Performing Test MFPU_NEON_AVAILABLE - Success
-- Performing Test FNO_LTO_AVAILABLE
-- Performing Test FNO_LTO_AVAILABLE - Success
-- Looking for sys/sdt.h
-- Looking for sys/sdt.h - not found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - done
-- Looking for fseeko
-- Looking for fseeko - found
-- Looking for strerror
-- Looking for strerror - found
-- Looking for posix_memalign
-- Looking for posix_memalign - found
-- Looking for aligned_alloc
-- Looking for aligned_alloc - found
-- Performing Test HAVE_NO_INTERPOSITION
-- Performing Test HAVE_NO_INTERPOSITION - Success
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN - Success
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL - Success
-- Performing Test HAVE_BUILTIN_CTZ
-- Performing Test HAVE_BUILTIN_CTZ - Success
-- Performing Test HAVE_BUILTIN_CTZLL
-- Performing Test HAVE_BUILTIN_CTZLL - Success
-- Performing Test HAVE_PTRDIFF_T
-- Performing Test HAVE_PTRDIFF_T - Success
-- Performing Test ARM_AUXV_HAS_CRC32
-- Performing Test ARM_AUXV_HAS_CRC32 - Failed
-- Performing Test ARM_HWCAP_HAS_CRC32
-- Performing Test ARM_HWCAP_HAS_CRC32 - Success
-- Performing Test ARM_AUXV_HAS_NEON
-- Performing Test ARM_AUXV_HAS_NEON - Success
-- Architecture-specific source files: arch/arm/armfeature.c;arch/arm/adler32_neon.c;arch/arm/chunkset_neon.c;arch/arm/slide_neon.c
-- Renaming
-- /home/pi/python-blosc2/blosc2/c-blosc2/internal-complibs/zlib-ng-2.0.7/zconf.h
-- to 'zconf.h.included' because this file is included with zlib
-- but CMake generates it automatically in the build directory.
-- The following features have been enabled:
- CMAKE_BUILD_TYPE, Build type: Release (selected)
- WITH_GZFILEOP, Compile with support for gzFile related functions
- ZLIB_COMPAT, Compile with zlib compatible API
- WITH_OPTIM, Build with optimisation
- WITH_NEW_STRATEGIES, Use new strategies
- WITH_UNALIGNED, Support unaligned reads on platforms that support it
- WITH_ACLE, Build with ACLE
- WITH_NEON, Build with NEON intrinsics
- NEON_ADLER32, Support NEON instructions in adler32, using "-mfpu=neon"
- NEON_SLIDEHASH, Support NEON instructions in slide_hash, using "-mfpu=neon"
-- The following OPTIONAL packages have been found:
- PythonLibs
-- The following REQUIRED packages have been found:
- PythonInterp
- Cython
- NumPy
- Threads
-- The following features have been disabled:
- ZLIB_ENABLE_TESTS, Build test binaries
- ZLIB_DUAL_LINK, Dual link tests against system zlib
- WITH_FUZZERS, Build test/fuzz
- WITH_NATIVE_INSTRUCTIONS, Instruct the compiler to use the full instruction set on this host (gcc/clang -march=native)
- WITH_MAINTAINER_WARNINGS, Build with project maintainer warnings
- WITH_CODE_COVERAGE, Enable code coverage reporting
- WITH_INFLATE_STRICT, Build with strict inflate distance checking
- WITH_INFLATE_ALLOW_INVALID_DIST, Build with zero fill for inflate invalid distances
- INSTALL_UTILS, Copy minigzip and minideflate during install
-- The following REQUIRED packages have not been found:
- PythonExtensions
-- Using ZSTD internal sources.
-- Building for system processor armv7l
-- Building for compiler ID GNU
-- Detected that BLOSC is used a subproject.
-- Disabling support for assembly sources in ZSTD
-- Adding run-time support for NEON
CMake Error at blosc2/CMakeLists.txt:25 (include_directories):
include_directories given empty-string as include directory.
cmake --version
cmake version 3.27.7
So, although Cmake found Numpy, it did not set
include_directories("${Python_NumPy_INCLUDE_DIRS}")
commenting this line out allowed compilation to proceed
I have tried that with recent C-Blosc2 2.11.2, and things looks good when compiling on Pi:
$ python3 setup.py build_ext -j 4 -i
<snip>
-- The following REQUIRED packages have been found:
* PythonInterp
* Cython
* NumPy
* Threads
<snip>
-- Installing: /root/python-blosc2/_skbuild/linux-armv7l-3.9/cmake-install/lib/pkgconfig/blosc2.pc
copying _skbuild/linux-armv7l-3.9/cmake-install/blosc2/blosc2_ext.cpython-39-arm-linux-gnueabihf.so -> blosc2/blosc2_ext.cpython-39-arm-linux-gnueabihf.so
running build_ext
The test suite is failing in a couple of places though.
test_lossy.pyfails with a Bus error:
$ PYTHONPATH=. pytest
<snip>
tests/ndarray/test_lossy.py ..Fatal Python error: Bus error
I suppose this is due to a an unaligned memory address somewhere in the lossy codecs (ZFP or elsewhere). Removing the test_lossy.py allows to progress further ahead:
- Two additional tests (softly) fail when trying to address more than 2 GB of memory:
$ PYTHONPATH=. pytest
========================================================= test session starts ==========================================================
platform linux -- Python 3.9.2, pytest-7.4.3, pluggy-1.3.0
rootdir: /root/python-blosc2
configfile: pytest.ini
testpaths: tests, blosc2/core.py, blosc2/ndarray.py, blosc2/schunk.py
collected 1384 items
tests/test_bytes_array.py ...... [ 0%]
tests/test_comp_info.py ..... [ 0%]
tests/test_compress2.py ........................ [ 2%]
tests/test_compression_parameters.py ... [ 2%]
tests/test_compressors.py ................................................................................ [ 8%]
tests/test_decompress.py ................ [ 9%]
tests/test_iterchunks.py ................................ [ 11%]
tests/test_open.py ........................... [ 13%]
tests/test_postfilters.py .................... [ 15%]
tests/test_prefilters.py ............................................ [ 18%]
tests/test_python_blosc.py ................... [ 19%]
tests/test_schunk.py .................................................................................................... [ 27%]
tests/test_schunk_constructor.py ................................ [ 29%]
tests/test_schunk_delete.py ................................ [ 31%]
tests/test_schunk_get_slice.py ......................................................... [ 35%]
tests/test_schunk_insert.py .................................................................................................... [ 43%]
................................................................................................................................ [ 52%]
............................ [ 54%]
tests/test_schunk_set_slice.py ......................................................... [ 58%]
tests/test_schunk_update.py .................................................................................................... [ 65%]
............................................................................................ [ 72%]
tests/test_tensor.py .......F...ssssss....F....ssss.. [ 74%]
tests/test_ucodecs.py ................ [ 75%]
tests/test_ufilters.py ................ [ 77%]
tests/test_vlmeta.py ........ [ 77%]
tests/ndarray/test_auto_parts.py ............................................................................................... [ 84%]
............................................. [ 87%]
tests/ndarray/test_buffer.py ........ [ 88%]
tests/ndarray/test_copy.py ............ [ 89%]
tests/ndarray/test_empty.py .......... [ 89%]
tests/ndarray/test_full.py ........ [ 90%]
tests/ndarray/test_getitem.py .......... [ 91%]
tests/ndarray/test_iterchunks_info.py .... [ 91%]
tests/ndarray/test_metalayers.py ...... [ 91%]
tests/ndarray/test_mode.py ...... [ 92%]
tests/ndarray/test_ndarray.py ................................ [ 94%]
tests/ndarray/test_numpy.py .. [ 94%]
tests/ndarray/test_persistency.py ....... [ 95%]
tests/ndarray/test_resize.py ... [ 95%]
tests/ndarray/test_setitem.py ... [ 95%]
tests/ndarray/test_slice.py ...... [ 96%]
tests/ndarray/test_squeeze.py .... [ 96%]
tests/ndarray/test_struct_dtype.py ..... [ 96%]
tests/ndarray/test_zeros.py ......... [ 97%]
blosc2/core.py .................. [ 98%]
blosc2/ndarray.py ............. [ 99%]
blosc2/schunk.py .... [100%]
=============================================================== FAILURES ===============================================================
_________________________________________________ test_pack_array2[300000000.0-int64] __________________________________________________
size = 300000000.0, dtype = 'int64'
@pytest.mark.parametrize(
"size, dtype",
[
(1e6, "int64"),
(1e6, "float64"),
(1e6, np.float64),
(1e6, np.int8),
pytest.param(3e8, "int64", marks=pytest.mark.heavy), # > 2 GB
],
)
def test_pack_array2(size, dtype):
> nparray = np.arange(size, dtype=dtype)
E ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size.
tests/test_tensor.py:49: ValueError
______________________________________________ test_pack_tensor_array[300000000.0-int64] _______________________________________________
size = 300000000.0, dtype = 'int64'
@pytest.mark.parametrize(
"size, dtype",
[
(1e6, "int64"),
(1e6, "float64"),
(1e6, np.float64),
(1e6, np.int8),
pytest.param(3e8, "int64", marks=pytest.mark.heavy), # > 2 GB
],
)
def test_pack_tensor_array(size, dtype):
> nparray = np.arange(size, dtype=dtype)
E ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size.
tests/test_tensor.py:123: ValueError
======================================================= short test summary info ========================================================
FAILED tests/test_tensor.py::test_pack_array2[300000000.0-int64] - ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size.
FAILED tests/test_tensor.py::test_pack_tensor_array[300000000.0-int64] - ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size.
======================================== 2 failed, 1372 passed, 10 skipped in 128.43s (0:02:08) ========================================
This is mainly due to the fact that Blosc2 is more meant to run on 64-bit platforms and 32-bit ones are no tested often (i.e. they are basically unsupported). For full support of 32-bit, better stay using Blosc 1.x series.