Is littler intended to be compatible with R CMD INSTALL?
ctheune opened this issue · 14 comments
Hi,
I'm not an R user but currently setting up an R environment for a user. We're using NixOS (Linux) which provides a number of automatically packaged R packages based on CRAN. Our builder at some point runs:
$rCommand CMD INSTALL $installFlags --configure-args="$configureFlags" -l $out/library .
+ R CMD INSTALL --configure-args= -l /nix/store/mgbk39lm6i7xhwx9pzjpynzj8fpfjqy3-r-littler-0.3.3/library .
This breaks when CMD INSTALL does its check thing:
building path(s) ‘/nix/store/mgbk39lm6i7xhwx9pzjpynzj8fpfjqy3-r-littler-0.3.3’
| executing builder ‘/nix/store/czx8vkrb9jdgjyz8qfksh10vrnqa723l-bash-4.4-p23/bin/bash’
unpacking sources
unpacking source archive /nix/store/w3ish8b9my5k60llc6qig3vw1b9ij9rg-littler_0.3.3.tar.gz
source root is littler
setting SOURCE_DATE_EPOCH to timestamp 1513529959 of file littler/MD5
patching sources
configuring
building
running tests
installing
* installing *source* package 'littler' ...
** package 'littler' successfully unpacked and MD5 sums checked
checking for gcc... /nix/store/iw94llkj05wgaz268mlzvgx8jkbi1ss0-gcc-wrapper-7.3.0/bin/cc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /nix/store/iw94llkj05wgaz268mlzvgx8jkbi1ss0-gcc-wrapper-7.3.0/bin/cc accepts -g... yes
checking for /nix/store/iw94llkj05wgaz268mlzvgx8jkbi1ss0-gcc-wrapper-7.3.0/bin/cc option to accept ISO C89... none needed
checking how to run the C preprocessor... /nix/store/iw94llkj05wgaz268mlzvgx8jkbi1ss0-gcc-wrapper-7.3.0/bin/cc -E
checking for grep that handles long lines and -e... /nix/store/9f89z51na7w931aja8lqlmhqny9h16cj-gnugrep-3.1/bin/grep
checking for egrep... /nix/store/9f89z51na7w931aja8lqlmhqny9h16cj-gnugrep-3.1/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking getopt.h usability... yes
checking getopt.h presence... yes
checking for getopt.h... yes
checking whether sys/types.h defines makedev... yes
checking for setenv... yes
checking for gettimeofday... yes
checking for time... yes
checking if R was built as a shared library... yes
checking for install_name_tool... no
configure: creating ./config.status
config.status: creating src/Makevars
config.status: creating src/gitversion.h
config.status: creating src/config.h
** libs
/nix/store/iw94llkj05wgaz268mlzvgx8jkbi1ss0-gcc-wrapper-7.3.0/bin/cc -o r -g -O2 -I/nix/store/ap80ym1qd41cbpk04makh9ja9k25cypc-R-3.5.1/lib/R/include littler.c -Wl,--export-dynamic -fopenmp -L/nix/store/ap80ym1qd41cbpk04makh9ja9k25cypc-R-3.5.1/lib/R/lib -lR -lpcre -llzma -lbz2 -lz -lrt -ldl -lm -licuuc -licui18n -L/nix/store/blk28p4cr6r2nc7fi1c4gggiqpd7pkqy-openblas-0.3.1/lib -lopenblas -L/nix/store/blk28p4cr6r2nc7fi1c4gggiqpd7pkqy-openblas-0.3.1/lib -lopenblas -Wl,-rpath,/nix/store/ap80ym1qd41cbpk04makh9ja9k25cypc-R-3.5.1/lib/R/lib -Wl,-rpath,/nix/store/m8bp0zw93ci3fqlcw33sm2vznfq2rv1w-gfortran-7.3.0-lib/lib -Wl,-rpath,/nix/store/mvfh8jj5200308mv2wms2bnwzy96yi3c-gfortran-wrapper-7.3.0/bin -Wl,-rpath,/nix/store/gyrx9n14b3b8jm8zqsya36zjkqk0pgvg-gfortran-7.3.0/lib/gcc/x86_64-unknown-linux-gnu/7.3.0 -Wl,-rpath,/nix/store/gyrx9n14b3b8jm8zqsya36zjkqk0pgvg-gfortran-7.3.0/lib -Wl,-rpath,/nix/store/blk28p4cr6r2nc7fi1c4gggiqpd7pkqy-openblas-0.3.1/lib -Wl,-rpath,/nix/store/pay7xbyp76hp6kcp6achl24dd9dinp8z-tcl-8.6.6/lib -Wl,-rpath,/nix/store/mkvyrnrzzlx00r66w3sqfw8zavgw0bn0-tk-8.6.6/lib -Wl,-rpath,/nix/store/fc3pvx20j5hh3jpf6skjzdf50p2pph15-openjdk-8u192b26-jre/lib/openjdk/jre/lib/amd64/server -Wl,-rpath,/nix/store/ap80ym1qd41cbpk04makh9ja9k25cypc-R-3.5.1/lib/R/lib -Wl,-rpath,/nix/store/m8bp0zw93ci3fqlcw33sm2vznfq2rv1w-gfortran-7.3.0-lib/lib -Wl,-rpath,/nix/store/mvfh8jj5200308mv2wms2bnwzy96yi3c-gfortran-wrapper-7.3.0/bin -Wl,-rpath,/nix/store/gyrx9n14b3b8jm8zqsya36zjkqk0pgvg-gfortran-7.3.0/lib/gcc/x86_64-unknown-linux-gnu/7.3.0 -Wl,-rpath,/nix/store/gyrx9n14b3b8jm8zqsya36zjkqk0pgvg-gfortran-7.3.0/lib -Wl,-rpath,/nix/store/blk28p4cr6r2nc7fi1c4gggiqpd7pkqy-openblas-0.3.1/lib -Wl,-rpath,/nix/store/pay7xbyp76hp6kcp6achl24dd9dinp8z-tcl-8.6.6/lib -Wl,-rpath,/nix/store/mkvyrnrzzlx00r66w3sqfw8zavgw0bn0-tk-8.6.6/lib -Wl,-rpath,/nix/store/fc3pvx20j5hh3jpf6skjzdf50p2pph15-openjdk-8u192b26-jre/lib/openjdk/jre/lib/amd64/server -Wl,-rpath,/nix/store/ap80ym1qd41cbpk04makh9ja9k25cypc-R-3.5.1/lib/R/lib -Wl,-rpath,/nix/store/m8bp0zw93ci3fqlcw33sm2vznfq2rv1w-gfortran-7.3.0-lib/lib -Wl,-rpath,/nix/store/mvfh8jj5200308mv2wms2bnwzy96yi3c-gfortran-wrapper-7.3.0/bin -Wl,-rpath,/nix/store/gyrx9n14b3b8jm8zqsya36zjkqk0pgvg-gfortran-7.3.0/lib/gcc/x86_64-unknown-linux-gnu/7.3.0 -Wl,-rpath,/nix/store/gyrx9n14b3b8jm8zqsya36zjkqk0pgvg-gfortran-7.3.0/lib -Wl,-rpath,/nix/store/blk28p4cr6r2nc7fi1c4gggiqpd7pkqy-openblas-0.3.1/lib -Wl,-rpath,/nix/store/pay7xbyp76hp6kcp6achl24dd9dinp8z-tcl-8.6.6/lib -Wl,-rpath,/nix/store/mkvyrnrzzlx00r66w3sqfw8zavgw0bn0-tk-8.6.6/lib -Wl,-rpath,/nix/store/fc3pvx20j5hh3jpf6skjzdf50p2pph15-openjdk-8u192b26-jre/lib/openjdk/jre/lib/amd64/server
*
* new binary r installed in bin/ subdirectory
* consider adding a symbolic link from, e.g., /usr/local/bin
* on OS X, you may have to name this 'lr' instead
*
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error: package or namespace load failed for 'littler':
.onAttach failed in attachNamespace() for 'littler', details:
call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
error: error in running command
Error: loading failed
Execution halted
ERROR: loading failed
* removing '/nix/store/mgbk39lm6i7xhwx9pzjpynzj8fpfjqy3-r-littler-0.3.3/library/littler'
note: keeping build directory ‘/tmp/nix-build-r-littler-0.3.3.drv-3’
builder for ‘/nix/store/f5hc9p8jp1jlibp1ksgng6ffyrn660xk-r-littler-0.3.3.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/xjm7kk53b61qhvpx1ndnrlzvbmp5rddp-R-3.5.1-wrapper.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/z5bx816hr47sizwdglv7yx4zmxlk4wa1-blackbee-wfe-1.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/z5bx816hr47sizwdglv7yx4zmxlk4wa1-blackbee-wfe-1.drv’ failed
Looking at the resulting R environment seems like it's working so far, but I guess this might be some incompatibility between the R CMD INSTALL invocation and running it with --no-test-load turns out fine.
So my question is: is this an build procedure thats supposed to be working and we have something weird in our setup or is this known to not work and running with --no-test-load is working as intended?
Can you rephrase your question and/or provide a reproducable example?
littler is on CRAN and as such is of course tested via / used with R CMD INSTALL etc.
edd@rob:~$ cd git/littler/
edd@rob:~/git/littler(master)$ R CMD INSTALL littler_0.3.5.tar.gz
* installing to library ‘/usr/local/lib/R/site-library’
* installing *source* package ‘littler’ ...
checking for gcc... ccache gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether ccache gcc accepts -g... yes
checking for ccache gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... ccache gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking getopt.h usability... yes
checking getopt.h presence... yes
checking for getopt.h... yes
checking whether sys/types.h defines makedev... yes
checking for setenv... yes
checking for gettimeofday... yes
checking for time... yes
checking if R was built as a shared library... yes
checking for install_name_tool... no
configure: creating ./config.status
config.status: creating src/Makevars
config.status: creating src/gitversion.h
config.status: creating src/config.h
** libs
ccache gcc -o r -g -O3 -Wall -pipe -pedantic -std=gnu99 -march=native -I/usr/share/R/include littler.c -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--export-dynamic -fopenmp -Wl,-Bsymbolic-functions -Wl,-z,relro -L/usr/lib/R/lib -lR -lpcre -llzma -lbz2 -lz -lrt -ldl -lm -licuuc -licui18n -lblas -llapack -Wl,-rpath,/usr/lib/R/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/jvm/default-java/lib/server -Wl,-rpath,/usr/lib/R/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/jvm/default-java/lib/server -Wl,-rpath,/usr/lib/R/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/jvm/default-java/lib/server
*
* new binary r installed in bin/ subdirectory
* consider adding a symbolic link from, e.g., /usr/local/bin
* on OS X, you may have to name this 'lr' instead
*
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (littler)
edd@rob:~/git/littler(master)$
```Thanks! That actually is already helpful to know: I'm not experienced in the R environment so I was pondering whether that is intended to work. This indicates that we'll have to fix something in our environment - fair enough!
Sure. I can possibly help with that with in "standard" setups it is tested regularly. Note though, and this is documented, that it requires R to be built as shared library to be embeddable. What you quoted in your first message was, and pardon the Hochdeutsch, rather "nutzlos" ;-) as all actual compile and link steps were omitted. This may be something simple, but as you say, likely on your side. If you have any local R users nearby maybe consult with them?
Hi,
yeah, we'll be in touch with a university colleague tomorrow. I'll update my pervious excerpt, but it looked clean so far.
Agreed. Apart from the non-standard path. R itself set them "in itself" (first level R is a shell script). littler tries to learn them at configure time and encode them. If your system is too non-standard, that fails. But yes, it all compiled (good), linked (good) and then created a non-loadable package. Which is very weird because littler actually as no R package in it - we use the package mechanism to have the r binary built.
Your error is on system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE) suggesting that maybe your shell is non-standard or something else.
I will still close this as no error in littler has been demonstrated -- and NixOS is not a know or supported OS variant we test or develop on. But you have access to NixOS so once you work something out feel free to propose a pull request.
Right. Thanks for listening. :)
Something that I wasn't able to parse mentally was the error message, so we found that the "which" binary was missing in this environment. My colleague is preparing various patches for the build environment and will provide NixOS with a PR ...
Thanks again!
Just to let you know: after making which available in the build environment everything is now working as intended.
Ok -- type -p is the usual (more robust) alternative to which and I should maybe consider switching to it,
No worries from my perspective. NixOS is a peculiar environment and only provides exactly what dependencies are specified even for "minor", individual builds. Someone who created the R support didn't consider or notice which missing. It's absolutely not a problem from our perspective that you decide to rely upon it. :)
And it looks like it is R: I just use Sys.which() from this startup function so you would have to talk to them to get it changed.
So R is providing which as an actual API function, not as a reflection, right? So that makes it a hard dependency based on the R standard library and not something a user might accidentally trigger by calling a reflection-based API to call binaries ...
Not what I said. I said and showed that I call an R function called Sys.which(). I suspect that its implementation uses which but I don't have time to look it up for you.
I have no idea what you mean by reflection. This isn't Java. It is just functions calling function. Turtles all the way down, as always.
Ah, misunderstood you there. Looked it up for good measure and it's looking up which during configure and then calls it just as expected.
What I meant about reflection: there sometimes are languages which use attribute names or function names as variables to help with some metaprogramming, so you could write (e.g. Python): shell.foo() which could be translated into os.system("foo") automatically. Those can be hard to debug as the function foo() won't be literally defined but shell could be a proxy.
Nevertheless - I'm all set and good. :)
Yes. R does something similar and takes inspiration from many POSIX and other functions and provides wrappers -- on all platforms if possible.