zstd.h: No such file or directory
Closed this issue · 2 comments
js2264 commented
Hi Mike @grimbough,
I noticed you added the ZSTD filter to rhdf5filters. It installs ok an a vanilla R 4.2.* with Bioc 3.16 (I tried with the docker image r-base:4.2.0). However, it fails on the new release R 4.3.0. I tried with vanilla R 4.3.0 installed from pre-compiled binaries or provided by rocker (docker image r-base:4.3.0). I can install the package after manually installing zstd with apt install libzstd-dev. Is it the intended behavior?
J
- Failing with R 4.3.0 / Bioc 3.17:
> BiocManager::install('rhdf5filters')
'getOption("repos")' replaces Bioconductor standard repositories, see
'help("repositories", package = "BiocManager")' for details.
Replacement repositories:
CRAN: https://cloud.r-project.org
Bioconductor version 3.17 (BiocManager 1.30.20), R 4.3.0 (2023-04-21)
Installing package(s) 'rhdf5filters'
trying URL 'https://bioconductor.org/packages/3.17/bioc/src/contrib/rhdf5filters_1.11.3.tar.gz'
Content type 'application/x-gzip' length 2959850 bytes (2.8 MB)
==================================================
downloaded 2.8 MB
* installing *source* package ‘rhdf5filters’ ...
** using staged installation
configuring the BLOSC filter...
x86_CPU=true
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking whether C compiler accepts -msse2... yes
checking whether C compiler accepts -mavx2... yes
checking for gcc __builtin_cpu_init function... (cached) true
checking for x86 sse2 instruction support... yes
checking for x86 avx2 instruction support... yes
configure: creating ./config.status
config.status: creating src/blosc/lib/blosc-1.20.1/Makefile
config.status: creating src/blosc/Makefile
configure: BZIP2_LIB=bzip2/libH5Zbz2.so
configure: BLOSC_LIB=blosc/libH5Zblosc.so
configure: LZF_LIB=lzf/libH5Zlzf.so
configure: VBZ_LIB=vbz/libH5Zvbz.so
configure: ZSTD_LIB=zstd/libH5Zzstd.so
configure: creating ./config.status
config.status: creating src/blosc/lib/blosc-1.20.1/Makefile
config.status: creating src/blosc/Makefile
config.status: creating src/Makevars
** libs
using C compiler: ‘gcc (Debian 12.2.0-14) 12.2.0’
make -C zstd
make[1]: Entering directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/zstd'
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/debug.c -o common/debug.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/entropy_common.c -o common/entropy_common.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/error_private.c -o common/error_private.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/fse_decompress.c -o common/fse_decompress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/pool.c -o common/pool.o -o common/pool.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/threading.c -o common/threading.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/xxhash.c -o common/xxhash.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/zstd_common.c -o common/zstd_common.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/fse_compress.c -o compress/fse_compress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/hist.c -o compress/hist.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/huf_compress.c -o compress/huf_compress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_compress.c -o compress/zstd_compress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_compress_literals.c -o compress/zstd_compress_literals.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_compress_sequences.c -o compress/zstd_compress_sequences.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_compress_superblock.c -o compress/zstd_compress_superblock.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_double_fast.c -o compress/zstd_double_fast.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_fast.c -o compress/zstd_fast.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_lazy.c -o compress/zstd_lazy.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_ldm.c -o compress/zstd_ldm.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_opt.c -o compress/zstd_opt.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstdmt_compress.c -o compress/zstdmt_compress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c decompress/huf_decompress.c -o decompress/huf_decompress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c decompress/zstd_ddict.c -o decompress/zstd_ddict.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c decompress/zstd_decompress.c -o decompress/zstd_decompress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c decompress/zstd_decompress_block.c -o decompress/zstd_decompress_block.o
rm -f libzstd.a
ar cq libzstd.a common/debug.o common/entropy_common.o common/error_private.o common/fse_decompress.o common/pool.o common/threading.o common/xxhash.o common/zstd_common.o compress/fse_compress.o compress/hist.o compress/huf_compress.o compress/zstd_compress.o compress/zstd_compress_literals.o compress/zstd_compress_sequences.o compress/zstd_compress_superblock.o compress/zstd_double_fast.o compress/zstd_fast.o compress/zstd_lazy.o compress/zstd_ldm.o compress/zstd_opt.o compress/zstdmt_compress.o decompress/huf_decompress.o decompress/zstd_ddict.o decompress/zstd_decompress.o decompress/zstd_decompress_block.o
ranlib libzstd.a
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -shared H5Zzstd.c -I"/usr/local/lib/R/site-library/Rhdf5lib/include" \
libzstd.a -Wl,-z,relro -o libH5Zzstd.so
make[1]: Leaving directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/zstd'
make -C blosc
make[1]: Entering directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc'
make -C lib/snappy-1.1.1
make[2]: Entering directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc/lib/snappy-1.1.1'
g++ -std=gnu++17 -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -c snappy.cc
g++ -std=gnu++17 -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -c snappy-c.cc
g++ -std=gnu++17 -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -c snappy-sinksource.cc
g++ -std=gnu++17 -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -c snappy-stubs-internal.cc
make[2]: Leaving directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc/lib/snappy-1.1.1'
make -C lib/lz4-1.9.4
make[2]: Entering directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc/lib/lz4-1.9.4'
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -c lz4.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -c lz4hc.c
make[2]: Leaving directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc/lib/lz4-1.9.4'
make -C lib/blosc-1.20.1 all
make[2]: Entering directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc/lib/blosc-1.20.1'
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c bitshuffle-avx2.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c shuffle-avx2.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c bitshuffle-sse2.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c shuffle-sse2.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c bitshuffle-generic.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c shuffle-generic.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -DSHUFFLE_SSE2_ENABLED -DSHUFFLE_AVX2_ENABLED -c shuffle.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c blosclz.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c blosc.c
blosc.c:37:12: fatal error: zstd.h: No such file or directory
37 | #include "zstd.h"
| ^~~~~~~~
compilation terminated.
make[2]: *** [Makefile:28: blosc.o] Error 1
make[2]: Leaving directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc/lib/blosc-1.20.1'
make[1]: *** [Makefile:39: blosc] Error 2
make[1]: Leaving directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc'
make: *** [Makevars:37: libH5Zblosc.so] Error 2
ERROR: compilation failed for package ‘rhdf5filters’
* removing ‘/usr/local/lib/R/site-library/rhdf5filters’
Session info:
R version 4.3.0 (2023-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 12 (bookworm)
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.21.so; LAPACK version 3.11.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
time zone: Etc/UTC
tzcode source: system (glibc)
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] BiocManager_1.30.20 compiler_4.3.0 tools_4.3.0
- Working with R 4.2.0 / Bioc 3.16:
R version 4.2.0 (2022-04-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux bookworm/sid
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] rhdf5filters_1.10.1
loaded via a namespace (and not attached):
[1] BiocManager_1.30.20 compiler_4.2.0 tools_4.2.0
grimbough commented
Hi @js2264 ,
Thanks for the report. I restructured the location of the bundled zstd library, but the intention was still to use that rather than relying on anything system level.
Does it work if you use the latest version from here? I guess you should remove the apt install libzstd-dev version first.
BiocManager::install('grimbough/rhdf5filters')js2264 commented
I checked on a fresh container without the pre-installed library and your changes fixed this issue, at least for the install step. Thanks for your fast reactivity!
J