How to install extra apt packages easily in biocorecrg-mopbasecall
eylenth opened this issue · 21 comments
Hello
A bioscientist is trying to execute the following command from the NanoPreprocess directory:
nextflow run nanopreprocess.nf -with-singularity
But then we see the following error
guppy_basecaller: error while loading shared libraries: libssl.so.10: cannot open shared object file: No such file or directory
which means that some apt packages are missing in the biocorecrg-mopbasecall-0.1 container
It seems the guppy_basecaller is missing the following libraries in the singularity container:
- libssl.so.10
- libcrypto.so.10
- libsz.so.2
(We are using guppy version 4.0.11 btw)
How can I easily install extra apt packages in biocorecrg-mopbasecall container ? I am not a nextflow export, so I don't know if there is even an 'easy' way to do this.
Hi, I don't know if we tested that version. Can you try with the version we described in the paper? If you still have the problem is likely another kind of problem.
@ADelgadoT did you tested the version 4.0.11?
I tested version 4.0.15 and it worked. I did not test 4.0.11.
I'm using the rpm installation to install guppy on my client server. I don't know if this matter?
Hi it seems to be working when I use the tarball of guppy instead of the rpm version.
But I also needed to modifiy the following file:
$MASTEROFPORESINSTALL/NanoPreprocess/nanopreprocess.nf
modified
gpu_prefix = 'export LD_LIBRARY_PATH="/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs"'
to
gpu_prefix = 'export LD_LIBRARY_PATH="/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs:${GUPPYSOFTAWAREPATH}/4.2.2/lib"'
so it could also read the guppy libraries inside the singularity container.
I don't understand. Have you placed the guppy folder inside bin in NanoPreprocess?
Like here:
ls ont-guppy_xx.xx.xx_linux64/ont-guppy/
bin data lib
you should have bin, data e lib folders. Then once you make links like here:
ls
guppy_aligner -> ont-guppy_xx.xx.xx_linux64/ont-guppy/bin/guppy_aligner
guppy_barcoder -> ont-guppy_xx.xx.xx_linux64/ont-guppy/bin/guppy_barcoder
guppy_basecaller -> ont-guppy_xx.xx.xx_linux64/ont-guppy/bin/guppy_basecaller
it should automatically mount the lib ...
It didn't work for me..
$ pwd
${MASTEROFPORESINSTALLDIR}/20201104-guppy-4.2.2/NanoPreprocess/bin
$ ls ont-guppy
bin data lib
$ ls -ls
total 128
0 -rwxr-xr-x 1 easybuild easybuild 3220 Dec 10 17:39 bam2stats.py
128 -rwxr-xr-x 1 easybuild easybuild 21907 Dec 10 17:39 deeplexicon.py
0 -rwxr-xr-x 1 easybuild easybuild 1117 Dec 10 17:39 extract_sequence_from_fastq.py
0 -rwxr-xr-x 1 easybuild easybuild 1416 Dec 10 17:39 fast5_to_fastq.py
0 -rwxr-xr-x 1 easybuild easybuild 642 Dec 10 17:39 fast5_type.py
0 lrwxrwxrwx 1 easybuild easybuild 27 Dec 10 17:41 guppy_aligner -> ont-guppy/bin/guppy_aligner
0 lrwxrwxrwx 1 easybuild easybuild 28 Dec 10 17:41 guppy_barcoder -> ont-guppy/bin/guppy_barcoder
0 lrwxrwxrwx 1 easybuild easybuild 35 Dec 10 17:41 guppy_basecall_client -> ont-guppy/bin/guppy_basecall_client
0 lrwxrwxrwx 1 easybuild easybuild 30 Dec 10 17:41 guppy_basecaller -> ont-guppy/bin/guppy_basecaller
0 lrwxrwxrwx 1 easybuild easybuild 41 Dec 10 17:41 guppy_basecaller_supervisor -> ont-guppy/bin/guppy_basecaller_supervisor
0 lrwxrwxrwx 1 easybuild easybuild 35 Dec 10 17:41 guppy_basecall_server -> ont-guppy/bin/guppy_basecall_server
0 drwxr-xr-x 5 easybuild easybuild 4096 Dec 10 14:53 ont-guppy
0 -rwxr-xr-x 1 easybuild easybuild 899 Dec 10 17:39 reorganize.sh
$ ldd guppy_barcoder
linux-vdso.so.1 => (0x00007fffdb3b0000)
libboost_thread.so.1.66.0 => not found
libboost_chrono.so.1.66.0 => not found
libboost_system.so.1.66.0 => not found
libboost_date_time.so.1.66.0 => not found
libboost_atomic.so.1.66.0 => not found
libz.so.1 => /lib64/libz.so.1 (0x00007fec8af31000)
libboost_filesystem.so.1.66.0 => not found
libboost_iostreams.so.1.66.0 => not found
libboost_log.so.1.66.0 => not found
libboost_log_setup.so.1.66.0 => not found
libboost_program_options.so.1.66.0 => not found
libboost_regex.so.1.66.0 => not found
libcuda.so.1 => /lib64/libcuda.so.1 (0x00007fec89ddb000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fec89bd7000)
librt.so.1 => /lib64/librt.so.1 (0x00007fec899cf000)
libm.so.6 => /lib64/libm.so.6 (0x00007fec896cd000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fec893c6000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fec891b0000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fec88f94000)
libc.so.6 => /lib64/libc.so.6 (0x00007fec88bc7000)
/lib64/ld-linux-x86-64.so.2 (0x00007fec8b147000)
libnvidia-fatbinaryloader.so.418.39 => /lib64/libnvidia-fatbinaryloader.so.418.39 (0x00007fec88979000)what is inside ont-guppy folder?
L
$ ls ont-guppy
bin data lib
and inside lib?
so the structure I have is:
master_of_pores/NanoPreprocess/bin/ont-guppy_3.1.5_linux64/ont-guppy/lib
is the same you have for the newer version?
$ ls -ls ont-guppy/lib/
total 20352
0 lrwxrwxrwx 1 easybuild easybuild 25 Sep 21 16:26 libboost_atomic.so -> libboost_atomic.so.1.66.0
128 -rw-r--r-- 1 easybuild easybuild 9019 Oct 22 2019 libboost_atomic.so.1.66.0
0 lrwxrwxrwx 1 easybuild easybuild 25 Sep 21 16:26 libboost_chrono.so -> libboost_chrono.so.1.66.0
128 -rw-r--r-- 1 easybuild easybuild 37603 Oct 22 2019 libboost_chrono.so.1.66.0
0 lrwxrwxrwx 1 easybuild easybuild 28 Sep 21 16:26 libboost_date_time.so -> libboost_date_time.so.1.66.0
128 -rw-r--r-- 1 easybuild easybuild 108155 Oct 22 2019 libboost_date_time.so.1.66.0
0 lrwxrwxrwx 1 easybuild easybuild 29 Sep 21 16:26 libboost_filesystem.so -> libboost_filesystem.so.1.66.0
128 -rw-r--r-- 1 easybuild easybuild 122153 Oct 22 2019 libboost_filesystem.so.1.66.0
0 lrwxrwxrwx 1 easybuild easybuild 28 Sep 21 16:26 libboost_iostreams.so -> libboost_iostreams.so.1.66.0
256 -rw-r--r-- 1 easybuild easybuild 139806 Oct 22 2019 libboost_iostreams.so.1.66.0
0 lrwxrwxrwx 1 easybuild easybuild 28 Sep 21 16:26 libboost_log_setup.so -> libboost_log_setup.so.1.66.0
1152 -rw-r--r-- 1 easybuild easybuild 1141501 Oct 22 2019 libboost_log_setup.so.1.66.0
0 lrwxrwxrwx 1 easybuild easybuild 22 Sep 21 16:26 libboost_log.so -> libboost_log.so.1.66.0
1280 -rw-r--r-- 1 easybuild easybuild 1218520 Oct 22 2019 libboost_log.so.1.66.0
0 lrwxrwxrwx 1 easybuild easybuild 34 Sep 21 16:26 libboost_program_options.so -> libboost_program_options.so.1.66.0
640 -rw-r--r-- 1 easybuild easybuild 554344 Oct 22 2019 libboost_program_options.so.1.66.0
0 lrwxrwxrwx 1 easybuild easybuild 24 Sep 21 16:26 libboost_regex.so -> libboost_regex.so.1.66.0
1280 -rw-r--r-- 1 easybuild easybuild 1272462 Oct 22 2019 libboost_regex.so.1.66.0
0 lrwxrwxrwx 1 easybuild easybuild 25 Sep 21 16:26 libboost_system.so -> libboost_system.so.1.66.0
128 -rw-r--r-- 1 easybuild easybuild 18513 Oct 22 2019 libboost_system.so.1.66.0
0 lrwxrwxrwx 1 easybuild easybuild 25 Sep 21 16:26 libboost_thread.so -> libboost_thread.so.1.66.0
256 -rw-r--r-- 1 easybuild easybuild 199538 Oct 22 2019 libboost_thread.so.1.66.0
0 lrwxrwxrwx 1 easybuild easybuild 19 Sep 21 16:26 libcrypto.so -> libcrypto.so.1.0.1e
0 lrwxrwxrwx 1 easybuild easybuild 19 Sep 21 16:26 libcrypto.so.10 -> libcrypto.so.1.0.1e
2048 -rw-r--r-- 1 easybuild easybuild 1971488 Aug 14 2019 libcrypto.so.1.0.1e
768 -rw-r--r-- 1 easybuild easybuild 751872 Oct 22 2019 libcurl.so
0 lrwxrwxrwx 1 easybuild easybuild 19 Sep 21 16:26 libhdf5_hl.so -> libhdf5_hl.so.8.0.0
384 -rw-r--r-- 1 easybuild easybuild 335623 Oct 22 2019 libhdf5_hl.so.1.8.11
0 lrwxrwxrwx 1 easybuild easybuild 20 Sep 21 16:26 libhdf5_hl.so.8.0.0 -> libhdf5_hl.so.1.8.11
0 lrwxrwxrwx 1 easybuild easybuild 16 Sep 21 16:26 libhdf5.so -> libhdf5.so.8.0.0
9344 -rw-r--r-- 1 easybuild easybuild 9487318 Oct 22 2019 libhdf5.so.1.8.11
0 lrwxrwxrwx 1 easybuild easybuild 17 Sep 21 16:26 libhdf5.so.8.0.0 -> libhdf5.so.1.8.11
0 lrwxrwxrwx 1 easybuild easybuild 25 Sep 21 16:26 libont_minimap2.so.2 -> libont_minimap2.so.2.17.0
256 -rw-r--r-- 1 easybuild easybuild 192368 Sep 21 16:12 libont_minimap2.so.2.17.0
0 lrwxrwxrwx 1 easybuild easybuild 21 Sep 21 16:26 libpgm-5.1.so.0 -> libpgm-5.1.so.0.0.118
384 -rw-r--r-- 1 easybuild easybuild 294848 Dec 22 2012 libpgm-5.1.so.0.0.118
0 lrwxrwxrwx 1 easybuild easybuild 16 Sep 21 16:26 libssl.so -> libssl.so.1.0.1e
0 lrwxrwxrwx 1 easybuild easybuild 16 Sep 21 16:26 libssl.so.10 -> libssl.so.1.0.1e
512 -rw-r--r-- 1 easybuild easybuild 443416 Aug 14 2019 libssl.so.1.0.1e
0 lrwxrwxrwx 1 easybuild easybuild 22 Sep 21 16:26 libvbz_hdf_plugin.so -> libvbz_hdf_plugin.so.1
0 lrwxrwxrwx 1 easybuild easybuild 26 Sep 21 16:26 libvbz_hdf_plugin.so.1 -> libvbz_hdf_plugin.so.1.0.0
640 -rwxr-xr-x 1 easybuild easybuild 591992 Sep 21 16:26 libvbz_hdf_plugin.so.1.0.0
0 lrwxrwxrwx 1 easybuild easybuild 15 Sep 21 16:26 libzmq.so -> libzmq.so.3.0.0
0 lrwxrwxrwx 1 easybuild easybuild 15 Sep 21 16:26 libzmq.so.3 -> libzmq.so.3.0.0
384 -rw-r--r-- 1 easybuild easybuild 316688 Feb 21 2015 libzmq.so.3.0.0
0 lrwxrwxrwx 1 easybuild easybuild 13 Dec 10 14:56 libz.so -> libz.so.1.2.3
0 lrwxrwxrwx 1 easybuild easybuild 13 Sep 21 16:26 libz.so.1 -> libz.so.1.2.3
128 -rw-r--r-- 1 easybuild easybuild 88600 Feb 22 2013 libz.so.1.2.3
0 -rw-r--r-- 1 easybuild easybuild 1235 Sep 21 16:12 MINIMAP2_LICENSE
my structure looks like this
ls NanoPreprocess/bin/ont-guppy/lib/
and We have guppy 4.2.2
ok so this is the problem I think. Make another folder named ont-guppy_4.2.2_linux64 and put ont-guppy inside to recreate the structure we have. Then make the soft links and try again
L
It won't work either:
$ ls -ls
total 128
-rwxr-xr-x 1 easybuild easybuild 3220 Dec 10 17:39 bam2stats.py
-rwxr-xr-x 1 easybuild easybuild 21907 Dec 10 17:39 deeplexicon.py
-rwxr-xr-x 1 easybuild easybuild 1117 Dec 10 17:39 extract_sequence_from_fastq.py
-rwxr-xr-x 1 easybuild easybuild 1416 Dec 10 17:39 fast5_to_fastq.py
-rwxr-xr-x 1 easybuild easybuild 642 Dec 10 17:39 fast5_type.py
lrwxrwxrwx 1 easybuild easybuild 51 Dec 10 18:03 guppy_aligner -> ont-guppy_4.2.2_linux64/ont-guppy/bin/guppy_aligner
lrwxrwxrwx 1 easybuild easybuild 52 Dec 10 18:03 guppy_barcoder -> ont-guppy_4.2.2_linux64/ont-guppy/bin/guppy_barcoder
lrwxrwxrwx 1 easybuild easybuild 59 Dec 10 18:03 guppy_basecall_client -> ont-guppy_4.2.2_linux64/ont-guppy/bin/guppy_basecall_client
lrwxrwxrwx 1 easybuild easybuild 54 Dec 10 18:03 guppy_basecaller -> ont-guppy_4.2.2_linux64/ont-guppy/bin/guppy_basecaller
lrwxrwxrwx 1 easybuild easybuild 65 Dec 10 18:03 guppy_basecaller_supervisor -> ont-guppy_4.2.2_linux64/ont-guppy/bin/guppy_basecaller_supervisor
lrwxrwxrwx 1 easybuild easybuild 59 Dec 10 18:03 guppy_basecall_server -> ont-guppy_4.2.2_linux64/ont-guppy/bin/guppy_basecall_server
drwxrwxr-x 3 easybuild easybuild 4096 Dec 10 18:03 ont-guppy_4.2.2_linux64
-rwxr-xr-x 1 easybuild easybuild 899 Dec 10 17:39 reorganize.sh
$ ldd guppy_barcoder
linux-vdso.so.1 => (0x00007ffcda121000)
libboost_thread.so.1.66.0 => not found
libboost_chrono.so.1.66.0 => not found
libboost_system.so.1.66.0 => not found
libboost_date_time.so.1.66.0 => not found
libboost_atomic.so.1.66.0 => not found
libz.so.1 => /lib64/libz.so.1 (0x00007f6cc5f1c000)
libboost_filesystem.so.1.66.0 => not found
libboost_iostreams.so.1.66.0 => not found
libboost_log.so.1.66.0 => not found
libboost_log_setup.so.1.66.0 => not found
libboost_program_options.so.1.66.0 => not found
libboost_regex.so.1.66.0 => not found
libcuda.so.1 => /lib64/libcuda.so.1 (0x00007f6cc4dc6000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f6cc4bc2000)
librt.so.1 => /lib64/librt.so.1 (0x00007f6cc49ba000)
libm.so.6 => /lib64/libm.so.6 (0x00007f6cc46b8000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f6cc43b1000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6cc419b000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6cc3f7f000)
libc.so.6 => /lib64/libc.so.6 (0x00007f6cc3bb2000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6cc6132000)
libnvidia-fatbinaryloader.so.418.39 => /lib64/libnvidia-fatbinaryloader.so.418.39 (0x00007f6cc3964000)
No but the missing libraries are in the container, so it is should be ok. Try to launch MoP. My ldd shows that these libraries are missing for me too.
Seems to be working with this procedure.
maybe can you clearly document on the installation procedure that you have to use the tarball of guppy and not the rpm one.
and copy the tarball to this exact location and not another location.
Hi, here the documentation about the installation. https://biocorecrg.github.io/master_of_pores/install.html
Feel free to suggest any modifications.
c) Guppy
Please note Guppy versions older than 3.1 (e.g. 3.0.3) only runs on CPUs. Newer versions (e.g. 3.1.5 and above) works on both CPUs and GPUs. The difference of speed between CPUs and GPU is more than 10 times.
Please do not use the rpm of Guppy, you have to use the tarball of Guppy.
cd master_of_pores/NanoPreprocess/bin
tar -zvxf ont-guppy_3.1.5_linux64.tar.gz #do not put the guppy tarball at another location.
ln -s ont-guppy_3.1.5_linux64/ont-guppy/bin/guppy_* .
Ok, I added a note. Thanks for this advice! f02a076