Problems during installation of dependencies

Question

Problems during installation of dependencies

Closed this issue 4 years ago · 6 comments

Hey Pay,
I am facing some problems during the installation of the dependencies with snakemake.
I received the following error message:

(nanopype) epicwl@epicwl-D3348-B2:~/pipelines/nanopype$ snakemake --snakefile rules/install.smk --directory ../.. all
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 Flappie
1 OpenBLAS
1 UCSCtools
1 all
1 bedtools
1 deepbinner
1 gitlfs
1 graphmap
1 hdf5
1 htslib
1 minimap2
1 nanopolish
1 ngmlr
1 samtools
1 sniffles
15

[Tue Feb 26 11:06:36 2019]
rule gitlfs:
input: /usr/bin
output: bin/git-lfs
jobid: 11

Fetching origin
remote: Enumerating objects: 36758, done.
remote: Counting objects: 100% (36758/36758), done.
remote: Compressing objects: 100% (10216/10216), done.
remote: Total 36272 (delta 25821), reused 35745 (delta 25309), pack-reused 0
Receiving objects: 100% (36272/36272), 9.48 MiB | 13.07 MiB/s, done.
Resolving deltas: 100% (25821/25821), completed with 394 local objects.
From https://github.com/git-lfs/git-lfs

[new tag] 2.0.2 -> 2.0.2
[new tag] v0.1.0 -> v0.1.0
[new tag] v0.2.0 -> v0.2.0
[new tag] v0.2.1 -> v0.2.1
[new tag] v0.2.1-p1 -> v0.2.1-p1
[new tag] v0.2.2 -> v0.2.2
[new tag] v0.2.3 -> v0.2.3
[new tag] v0.3.0 -> v0.3.0
[new tag] v0.3.1 -> v0.3.1
[new tag] v0.3.2 -> v0.3.2
[new tag] v0.3.3 -> v0.3.3
[new tag] v0.3.4 -> v0.3.4
[new tag] v0.3.5 -> v0.3.5
[new tag] v0.3.6 -> v0.3.6
[new tag] v0.4.0 -> v0.4.0
[new tag] v0.4.1 -> v0.4.1
[new tag] v0.4.2 -> v0.4.2
[new tag] v0.4.2.1 -> v0.4.2.1
[new tag] v0.5.0 -> v0.5.0
[new tag] v0.5.0.pre1 -> v0.5.0.pre1
[new tag] v0.5.0.pre2 -> v0.5.0.pre2
[new tag] v0.5.1 -> v0.5.1
[new tag] v0.5.1-tracing -> v0.5.1-tracing
[new tag] v0.5.2 -> v0.5.2
[new tag] v0.5.3 -> v0.5.3
[new tag] v0.5.4 -> v0.5.4
[new tag] v0.5.4-homebrew -> v0.5.4-homebrew
[new tag] v0.6.0 -> v0.6.0
[new tag] v1.0.0 -> v1.0.0
[new tag] v1.0.1 -> v1.0.1
[new tag] v1.0.2 -> v1.0.2
[new tag] v1.1.0 -> v1.1.0
[new tag] v1.1.1 -> v1.1.1
[new tag] v1.1.1-pre-push-tracing -> v1.1.1-pre-push-tracing
[new tag] v1.1.2 -> v1.1.2
[new tag] v1.2.0 -> v1.2.0
[new tag] v1.2.1 -> v1.2.1
[new tag] v1.3.0 -> v1.3.0
[new tag] v1.3.1 -> v1.3.1
[new tag] v1.4.0 -> v1.4.0
[new tag] v1.4.1 -> v1.4.1
[new tag] v1.4.2 -> v1.4.2
[new tag] v1.4.3 -> v1.4.3
[new tag] v1.4.4 -> v1.4.4
[new tag] v1.5.0 -> v1.5.0
[new tag] v1.5.1 -> v1.5.1
[new tag] v1.5.2 -> v1.5.2
[new tag] v1.5.2-go1.8beta1 -> v1.5.2-go1.8beta1
[new tag] v1.5.3 -> v1.5.3
[new tag] v1.5.4 -> v1.5.4
[new tag] v1.5.5 -> v1.5.5
[new tag] v1.5.6 -> v1.5.6
[new tag] v2.0.0 -> v2.0.0
[new tag] v2.0.1 -> v2.0.1
[new tag] v2.0.2 -> v2.0.2
[new tag] v2.0.2-rc1 -> v2.0.2-rc1
[new tag] v2.1.0 -> v2.1.0
[new tag] v2.1.1 -> v2.1.1
[new tag] v2.2.0 -> v2.2.0
[new tag] v2.2.1 -> v2.2.1
[new tag] v2.2.2-example -> v2.2.2-example
[new tag] v2.3.0 -> v2.3.0
[new tag] v2.3.1 -> v2.3.1
[new tag] v2.3.2 -> v2.3.2
[new tag] v2.3.3 -> v2.3.3
[new tag] v2.3.4 -> v2.3.4
[new tag] v2.4.0 -> v2.4.0
[new tag] v2.4.1 -> v2.4.1
[new tag] v2.4.2 -> v2.4.2
[new tag] v2.5.0 -> v2.5.0
[new tag] v2.5.1 -> v2.5.1
[new tag] v2.5.2 -> v2.5.2
[new tag] v2.6.1 -> v2.6.1
[new tag] v2.7.0 -> v2.7.0
HEAD is now at 93702d75 Merge pull request #3355 from bk2204/bump-version-2.6.0
can't load package: package github.com/git-lfs/git-lfs/commands: cannot find package "github.com/git-lfs/git-lfs/commands" in any of:
/usr/lib/go-1.10/src/github.com/git-lfs/git-lfs/commands (from $GOROOT)
/home/epicwl/go/src/github.com/git-lfs/git-lfs/commands (from $GOPATH)
template: main:1:31: executing "main" at <.Module>: can't evaluate field Module in type *load.PackagePublic
git-lfs: skipping fmt, no goimports found at goimports ...
go generate github.com/git-lfs/git-lfs/commands
can't load package: package github.com/git-lfs/git-lfs/commands: cannot find package "github.com/git-lfs/git-lfs/commands" in any of:
/usr/lib/go-1.10/src/github.com/git-lfs/git-lfs/commands (from $GOROOT)
/home/epicwl/go/src/github.com/git-lfs/git-lfs/commands (from $GOPATH)
Makefile:160: recipe for target 'commands/mancontent_gen.go' failed
make: *** [commands/mancontent_gen.go] Error 1
[Tue Feb 26 11:06:40 2019]
Error in rule gitlfs:
jobid: 11
output: bin/git-lfs

RuleException:
CalledProcessError in line 247 of /home/epicwl/pipelines/nanopype/rules/install.smk:
Command 'set -euo pipefail; export PATH=$(pwd)//usr/bin:$PATH
mkdir -p src && cd src
if [ ! -d git-lfs ]; then
git clone https://github.com/git-lfs/git-lfs.git --branch v2.6.0 --depth=1 && cd git-lfs
else
cd git-lfs && git fetch --all --tags --prune && git checkout tags/v2.6.0
fi
make
cp bin/git-lfs ../../bin' returned non-zero exit status 2.
File "/home/epicwl/pipelines/nanopype/rules/install.smk", line 247, in __rule_gitlfs
File "/home/epicwl/miniconda3/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/epicwl/.snakemake/log/2019-02-26T110636.098902.snakemake.log

I think it could be due to some of the github links....

Thanks for your support.
Best,

Joschka

Answer 1 · 2019-03-02T18:10:04.000Z

Hey Joschka,
seems like Go is not finding the git-lfs repo, I'll find a way to add it to GOPATH and patch it into the next master release.
In the mean time you could try the following (in the installation folder, I guess your home ~/)

mkdir -p src/gocode
export GOPATH=$(pwd)/src/gocode
/usr/bin/go get github.com/github/git-lfs
cp src/gocode/bin/git-lfs bin/

This should get you git-lfs and if you then re-run the install pipeline, the rule for Flappie should detect git-lfs present in bin/

Please let me know if this helps and if you have further trouble getting started.

Pay

Answer 2 · 2019-03-03T17:33:23.000Z

Hey Pay,
Thank you for the fix. It has worked!
However I am facing now a different problem during the installation of flappie. I has problems compiling the test_runner.

(nanopype) epicwl@epicwl-D3348-B2:~/pipelines/nanopype$ snakemake --snakefile rules/install.smk --directory ../.. all -p
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 Flappie
1 all
1 bedtools
1 deepbinner
1 graphmap
1 minimap2
1 ngmlr
7

[Sun Mar 3 19:28:57 2019]
rule Flappie:
input: bin/git-lfs, src/OpenBLAS, src/hdf5
output: bin/flappie
jobid: 1

    install_prefix=`pwd`
    bin/git-lfs install
    export PATH=$install_prefix/bin:$PATH
    mkdir -p src && cd src
    if [ ! -d flappie ]; then
        git clone https://github.com/nanoporetech/flappie --branch master && cd flappie
    else
        cd flappie && git fetch --all --tags --prune && git checkout master
    fi
    mkdir -p build && cd build && rm -rf * && cmake -DCMAKE_BUILD_TYPE=Release -DOPENBLAS_ROOT=$install_prefix -DHDF5_ROOT=$install_prefix -G"Unix Makefiles" ../
    cmake --build . --config Release -- -j 1
    cp flappie ../../../bin/flappie

Git LFS initialized.
Fetching origin
Already on 'master'
Your branch is up to date with 'origin/master'.
-- The C compiler identification is GNU 7.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Looking for hdf5.h
-- Looking for hdf5.h - found
-- Configuring done
-- Generating done
-- Build files have been written to: /home/epicwl/src/flappie/build
Scanning dependencies of target flappie_objects
[ 4%] Building C object CMakeFiles/flappie_objects.dir/src/decode.c.o
[ 8%] Building C object CMakeFiles/flappie_objects.dir/src/layers.c.o
[ 13%] Building C object CMakeFiles/flappie_objects.dir/src/networks.c.o
[ 17%] Building C object CMakeFiles/flappie_objects.dir/src/nnfeatures.c.o
[ 21%] Building C object CMakeFiles/flappie_objects.dir/src/flappie_common.c.o
[ 26%] Building C object CMakeFiles/flappie_objects.dir/src/flappie_matrix.c.o
/home/epicwl/src/flappie/src/flappie_matrix.c: In function ‘validate_flappie_matrix’:
/home/epicwl/src/flappie/src/flappie_matrix.c:150:45: warning: unused parameter ‘mat’ [-Wunused-parameter]
bool validate_flappie_matrix(flappie_matrix mat, float lower,
^~~
/home/epicwl/src/flappie/src/flappie_matrix.c:150:56: warning: unused parameter ‘lower’ [-Wunused-parameter]
bool validate_flappie_matrix(flappie_matrix mat, float lower,
^~~~~
/home/epicwl/src/flappie/src/flappie_matrix.c:151:43: warning: unused parameter ‘upper’ [-Wunused-parameter]
const float upper, const float maskval,
^~~~~
/home/epicwl/src/flappie/src/flappie_matrix.c:151:62: warning: unused parameter ‘maskval’ [-Wunused-parameter]
const float upper, const float maskval,
^~~~~~~
/home/epicwl/src/flappie/src/flappie_matrix.c:152:42: warning: unused parameter ‘only_finite’ [-Wunused-parameter]
const bool only_finite, const char *file,
^~~~~~~~~~~
/home/epicwl/src/flappie/src/flappie_matrix.c:152:67: warning: unused parameter ‘file’ [-Wunused-parameter]
const bool only_finite, const char *file,
^~~~
/home/epicwl/src/flappie/src/flappie_matrix.c:153:41: warning: unused parameter ‘line’ [-Wunused-parameter]
const int line) {
^~~~
/home/epicwl/src/flappie/src/flappie_matrix.c: In function ‘validate_vector’:
/home/epicwl/src/flappie/src/flappie_matrix.c:548:29: warning: unused parameter ‘vec’ [-Wunused-parameter]
bool validate_vector(float *vec, const size_t n, const float lower,
^~~
/home/epicwl/src/flappie/src/flappie_matrix.c:548:47: warning: unused parameter ‘n’ [-Wunused-parameter]
bool validate_vector(float *vec, const size_t n, const float lower,
^
/home/epicwl/src/flappie/src/flappie_matrix.c:548:62: warning: unused parameter ‘lower’ [-Wunused-parameter]
bool validate_vector(float *vec, const size_t n, const float lower,
^~~~~
/home/epicwl/src/flappie/src/flappie_matrix.c:549:34: warning: unused parameter ‘upper’ [-Wunused-parameter]
const float upper, const char *file, const int line) {
^~~~~
/home/epicwl/src/flappie/src/flappie_matrix.c:549:53: warning: unused parameter ‘file’ [-Wunused-parameter]
const float upper, const char *file, const int line) {
^~~~
/home/epicwl/src/flappie/src/flappie_matrix.c:549:69: warning: unused parameter ‘line’ [-Wunused-parameter]
const float upper, const char *file, const int line) {
^~~~
/home/epicwl/src/flappie/src/flappie_matrix.c: In function ‘validate_ivector’:
/home/epicwl/src/flappie/src/flappie_matrix.c:582:28: warning: unused parameter ‘vec’ [-Wunused-parameter]
bool validate_ivector(int *vec, const size_t n, const int lower, const int upper,
^~~
/home/epicwl/src/flappie/src/flappie_matrix.c:582:46: warning: unused parameter ‘n’ [-Wunused-parameter]
bool validate_ivector(int *vec, const size_t n, const int lower, const int upper,
^
/home/epicwl/src/flappie/src/flappie_matrix.c:582:59: warning: unused parameter ‘lower’ [-Wunused-parameter]
bool validate_ivector(int *vec, const size_t n, const int lower, const int upper,
^~~~~
/home/epicwl/src/flappie/src/flappie_matrix.c:582:76: warning: unused parameter ‘upper’ [-Wunused-parameter]
bool validate_ivector(int *vec, const size_t n, const int lower, const int upper,
^~~~~
/home/epicwl/src/flappie/src/flappie_matrix.c:583:35: warning: unused parameter ‘file’ [-Wunused-parameter]
const char *file, const int line) {
^~~~
/home/epicwl/src/flappie/src/flappie_matrix.c:583:51: warning: unused parameter ‘line’ [-Wunused-parameter]
const char *file, const int line) {
^~~~
[ 30%] Building C object CMakeFiles/flappie_objects.dir/src/flappie_output.c.o
[ 34%] Building C object CMakeFiles/flappie_objects.dir/src/flappie_structures.c.o
[ 39%] Building C object CMakeFiles/flappie_objects.dir/src/util.c.o
[ 39%] Built target flappie_objects
Scanning dependencies of target flappie_static
[ 43%] Linking C static library libflappie.a
[ 43%] Built target flappie_static
Scanning dependencies of target flappie
[ 47%] Building C object CMakeFiles/flappie.dir/src/fast5_interface.c.o
/home/epicwl/src/flappie/src/fast5_interface.c: In function ‘write_summary’:
/home/epicwl/src/flappie/src/fast5_interface.c:341:16: warning: unused variable ‘status2’ [-Wunused-variable]
herr_t status2 = write_trace(read_group, trace_flat, res.trace->nr, res.trace->nc,
^~~~~~~
/home/epicwl/src/flappie/src/fast5_interface.c:336:12: warning: unused variable ‘status’ [-Wunused-variable]
herr_t status = write_signal(read_group, res.rt.raw + res.rt.start,
^~~~~~
[ 52%] Building C object CMakeFiles/flappie.dir/src/flappie.c.o
[ 56%] Linking C executable flappie
[ 56%] Built target flappie
Scanning dependencies of target flappie_unittest
[ 60%] Building C object CMakeFiles/flappie_unittest.dir/src/test/flappie_test_runner.c.o
/home/epicwl/src/flappie/src/test/flappie_test_runner.c:11:10: fatal error: CUnit/Basic.h: No such file or directory
#include <CUnit/Basic.h>
^~~~~~~~~~~~~~~
compilation terminated.
CMakeFiles/flappie_unittest.dir/build.make:62: recipe for target 'CMakeFiles/flappie_unittest.dir/src/test/flappie_test_runner.c.o' failed
make[2]: *** [CMakeFiles/flappie_unittest.dir/src/test/flappie_test_runner.c.o] Error 1
CMakeFiles/Makefile2:178: recipe for target 'CMakeFiles/flappie_unittest.dir/all' failed
make[1]: *** [CMakeFiles/flappie_unittest.dir/all] Error 2
Makefile:162: recipe for target 'all' failed
make: *** [all] Error 2
[Sun Mar 3 19:29:26 2019]
Error in rule Flappie:
jobid: 1
output: bin/flappie

RuleException:
CalledProcessError in line 303 of /home/epicwl/pipelines/nanopype/rules/install.smk:
Command 'set -euo pipefail; install_prefix=pwd
bin/git-lfs install
export PATH=$install_prefix/bin:$PATH
mkdir -p src && cd src
if [ ! -d flappie ]; then
git clone https://github.com/nanoporetech/flappie --branch master && cd flappie
else
cd flappie && git fetch --all --tags --prune && git checkout master
fi
mkdir -p build && cd build && rm -rf * && cmake -DCMAKE_BUILD_TYPE=Release -DOPENBLAS_ROOT=$install_prefix -DHDF5_ROOT=$install_prefix -G"Unix Makefiles" ../
cmake --build . --config Release -- -j 1
cp flappie ../../../bin/flappie' returned non-zero exit status 2.
File "/home/epicwl/pipelines/nanopype/rules/install.smk", line 303, in __rule_Flappie
File "/home/epicwl/miniconda3/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/epicwl/.snakemake/log/2019-03-03T192857.783814.snakemake.log
(nanopype) epicwl@epicwl-D3348-B2:~/pipelines/nanopype$

Looking forward to testing your pipeline and thanks for he support
Best,
Joschka

Answer 3 · 2019-03-03T17:44:17.000Z

I was able to compile it with make. I guess this was not your original plan but it has worked. Will try analyzing some data soon!
Thanks,
Joschka

Answer 4 · 2019-03-05T09:22:10.000Z

Flappie needs libcunit1-dev to compile the unit tests. I wonder how make worked in you case and cmake doesn't. Have you any details for me?
I'll check if we can build Flappie without tests, to save the pipeline from that dependency.
Thanks again for testing.

Pay

Regarding analysis, please note we do not yet support the bulk fast5 which are the output from the updated MinKNOW. It is likely that if you sequenced in the past two weeks you can't yet use the pipeline. I'm fixing that today, stay tuned ;)

Answer 5 · 2019-03-06T07:58:42.000Z

I used the default installation of flappie as described on their github. Will install libcunit1-dev for further updates.
Thanks for the information. Currently we haven't done any sequencing yet. I guess we will start next week.
Best,
Joschka

Answer 6 · 2020-07-28T13:03:11.000Z

Closing this due to inactivity.
I assume it to be fixed as Flappie is successfully build in current singularity images.

Feel free to re-open if the error persists.