Problems building the bitshuffle and lz4 libraries on Centos7.
Closed this issue · 8 comments
I am having problems building the bitshuffle and lz4 libraries on Centos 7.
With bitshuffle there is no Makefile, there is a setup.py file. When I try to run that I get the following:
[root@corvette bitshuffle]# python setup.py install
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Unable to find pgen, not compiling formal grammar.
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
Compiling /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Plex/Scanners.py because it changed.
Compiling /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Plex/Actions.py because it changed.
Compiling /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/Pythran.py because it changed.
Compiling /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/Lexicon.py because it changed.
Compiling /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/Scanning.py because it changed.
Compiling /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/Parsing.py because it changed.
Compiling /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/Visitor.py because it changed.
Compiling /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/FlowControl.py because it changed.
Compiling /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/Code.py because it changed.
Compiling /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Runtime/refnanny.pyx because it changed.
Compiling /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/FusedNode.py because it changed.
Compiling /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Tempita/_tempita.py because it changed.
Compiling /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/StringIOTree.py because it changed.
[ 1/13] Cythonizing /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/Code.py
[ 2/13] Cythonizing /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/FlowControl.py
[ 3/13] Cythonizing /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/FusedNode.py
[ 4/13] Cythonizing /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/Lexicon.py
[ 5/13] Cythonizing /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/Parsing.py
[ 6/13] Cythonizing /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/Pythran.py
[ 7/13] Cythonizing /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/Scanning.py
[ 8/13] Cythonizing /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Compiler/Visitor.py
[ 9/13] Cythonizing /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Plex/Actions.py
[10/13] Cythonizing /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Plex/Scanners.py
[11/13] Cythonizing /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Runtime/refnanny.pyx
[12/13] Cythonizing /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/StringIOTree.py
[13/13] Cythonizing /tmp/easy_install-b0Y5TM/Cython-0.28.5/Cython/Tempita/_tempita.py
warning: no files found matching '2to3-fixers.txt'
warning: no files found matching 'Doc/*'
warning: no files found matching '*.pyx' under directory 'Cython/Debugger/Tests'
warning: no files found matching '*.pxd' under directory 'Cython/Debugger/Tests'
warning: no files found matching '*.pxd' under directory 'Cython/Utility'
warning: no files found matching 'pyximport/README'
Installed /usr/local/bitshuffle/Cython-0.28.5-py2.7-linux-x86_64.egg
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/bitshuffle
copying bitshuffle/__init__.py -> build/lib.linux-x86_64-2.7/bitshuffle
creating build/lib.linux-x86_64-2.7/bitshuffle/tests
copying bitshuffle/tests/__init__.py -> build/lib.linux-x86_64-2.7/bitshuffle/tests
copying bitshuffle/tests/make_regression_tdata.py -> build/lib.linux-x86_64-2.7/bitshuffle/tests
copying bitshuffle/tests/test_ext.py -> build/lib.linux-x86_64-2.7/bitshuffle/tests
copying bitshuffle/tests/test_h5filter.py -> build/lib.linux-x86_64-2.7/bitshuffle/tests
copying bitshuffle/tests/test_h5plugin.py -> build/lib.linux-x86_64-2.7/bitshuffle/tests
copying bitshuffle/tests/test_regression.py -> build/lib.linux-x86_64-2.7/bitshuffle/tests
creating build/lib.linux-x86_64-2.7/bitshuffle/tests/data
copying bitshuffle/tests/data/regression_0.1.3.h5 -> build/lib.linux-x86_64-2.7/bitshuffle/tests/data
running build_ext
#################################
# Compiling with OpenMP support #
#################################
building 'bitshuffle.ext' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/bitshuffle
creating build/temp.linux-x86_64-2.7/src
creating build/temp.linux-x86_64-2.7/lz4
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DBSHUF_VERSION_MAJOR=0 -DBSHUF_VERSION_MINOR=3 -DBSHUF_VERSION_POINT=5 -Isrc/ -Ilz4/ -I/usr/lib64/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -c bitshuffle/ext.c -o build/temp.linux-x86_64-2.7/bitshuffle/ext.o -O3 -ffast-math -march=native -std=c99 -fno-strict-aliasing -fopenmp
gcc: error: bitshuffle/ext.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 4
I downloaded the new version of lz4 here:
https://github.com/nexusformat/HDF5-External-Filter-Plugins
The README says
Currently the plugins must be built directly from the sources. On Windows
useCMakeand on Linux theautotoolsscripts delivered to build the code.
However, there are actually no autotools scripts provided. It appears that only cmake is supported. I tried to build with cmake, but it fails because Centos 7 has cmake 2.8.12.2, but the distribution requires cmake 3.0.0 or higher.
[root@corvette build]# cmake ..
CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
CMake 3.0.0 or higher is required. You are running version 2.8.12.2
I'm stuck.
ADSupport already contains the bitshuffle and lz4 codes. Would it be possible to have ADEiger use this? Currently those are built into libblosc.so. Can that be used, or do we need to make separate libraries for bitshuffle and lz4?
Hi Mark,
I honestly don't know. I remember building the "legacy" version of LZ4 and then never having to rebuild it again. FMX and AMX didn't open the HDF5 files with ADEiger, just used it to control the detector and then used third-party software to deal with the files.
Can that be used, or do we need to make separate libraries for bitshuffle and lz4?
You can test that by setting the HDF5_PLUGIN_PATH to the path where they reside and trying to open HDF5 files encoded with bitshuffle and lz4 with another tool, maybe hdfview.
but the distribution requires cmake 3.0.0 or higher
Maybe you can cheat and modify CMakeLists.txt to accept a lower version?
You can test that by setting the HDF5_PLUGIN_PATH to the path where they reside and trying to open HDF5 files encoded with bitshuffle and lz4 with another tool, maybe hdfview.**
HDF5_PLUGIN_PATH is only used to specify a directory, not a library name, right? How does the HDF5 library determine the name of the library to look for in that path? I don't think this is going to work, because I think that code registers the blosc compressor, which has a different ID from the lz4 and BitShuuffle compressors? I will give it a try though.
Maybe you can cheat and modify CMakeLists.txt to accept a lower version?
I tried that, but then I just get a whole bunch of cmake errors. I could try installing a later version of cmake, but I would rather use ADSupport if possible. That way people using old OS (Centos 6) can run out of the box.
I set the HDF5_PLUGIN_PATH to point to the location of the blosc filter:
corvette:CARS/iocBoot/ioc13Eiger1>ls $HDF5_PLUGIN_PATH
libMagick++.a libNeXus.a libbzlib.a libfilters.a libhdf5_hl.a libjp2.a liblcms.a libpng.a libtiff.a libwebp.a libxml2.a
libMagick++.so libNeXus.so libbzlib.so libfilters.so libhdf5_hl.so libjp2.so liblcms.so libpng.so libtiff.so libwebp.so libxml2.so
libMagick.a libblosc.a libcoders.a libhdf5.a libjbig.a libjpeg.a libnetCDF.a libszip.a libttf.a libwmf.a libzlib.a
libMagick.so libblosc.so libcoders.so libhdf5.so libjbig.so libjpeg.so libnetCDF.so libszip.so libttf.so libwmf.so libzlib.so
corvette:CARS/iocBoot/ioc13Eiger1>
It did not work, when I use the FileWriter as the data source and save LZ4 compressed files I get this error from ADEiger:
HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 140167793805056:
#000: ../H5Dio.c line 171 in H5Dread(): can't read data
major: Dataset
minor: Read failed
#001: ../H5Dio.c line 544 in H5D__read(): can't read data
major: Dataset
minor: Read failed
#002: ../H5Dchunk.c line 2050 in H5D__chunk_read(): unable to read raw data chunk
major: Low-level I/O
minor: Read failed
#003: ../H5Dchunk.c line 3405 in H5D__chunk_lock(): data pipeline read failed
major: Data filters
minor: Filter operation failed
#004: ../H5Z.c line 1357 in H5Z_pipeline(): required filter 'HDF5 lz4 filter; see http://www.hdfgroup.org/services/contributions.html' is not registered
major: Data filters
minor: Read failed
2018/08/28 08:46:58.047 eigerDetector::parseH5File: couldn't read image
My guess is that the Eiger is not using BLOSC. Then you were correct when you said:
because I think that code registers the blosc compressor, which has a different ID from the lz4 and BitShuuffle
Maybe the way forward would be to compile it with CMake 3 first, then try to bring that repository under ADSupport and develop custom vanilla Makefiles
Once CMake is working, perhaps this [1] will work too? Then the Makefiles will be ready to be included in ADSupport.
Xiaoqiang said that the needed code is already present in the bloscSrc directory in ADSupport. I just need to know how to package it so that HDF5 can use it. I am working on this approach.
I am closing this because we can use the code in ADSupport. This will use the EPICS build system and so my problems above will be moot.