deeplexicon in NanoPreproessSimple
yasukoh opened this issue · 17 comments
Hi,
I've been trying to use NanoPreprocessSimple since I had basecalled data already.
However, everytime I tried, I got an error saying " StreamExecutor device (0): ,
/usr/local/lib/python3.7/site-packages/pandas/compat/init.py:117: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError."
By searching the problem, I found the people says this is the problem of pandas and downgrade version will work or
install "xz" will work. Both tried, but did not dissolve this situation.
More puzzling thing is deeplexicon itself works on my system (mac OS Catalina, python is from anaconda3).
So I am wondering what could be done.
If you could help me with, I greatly appreciate it.
Thank you so much...
yh
Hi, we recommend to use MoP with singularity or docker images for overcoming this kind of issues.
Hi,
thank you for your comment and recommendation! I am afraid this might be very basic question.
But I'm sorry that I am not familiar with MoP.
would you explain to me a bit more? I am using with docker for NanoPreprocessSimple.
Thank you so much.
yh
So you are using docker and get this kind of error? Can you write down the command line you are using for launching nanopreprocess, please?
command line is
"nextflow run nanopreprocess_s.nf -with-docker"
yh
Hi, it looks like in Mac you need to mount /Users directory in docker:
https://www.nextflow.io/docs/latest/docker.html#
You might want to try singularity too:
https://sylabs.io/singularity-desktop-macos/
L
Hi,
The link you provided is a bit old info and with current docker, /Users directory is automatically mounted.
But I realized that docker tries to search python library which doesn't exist. So I tried to solve this.
With my knowledge I couldn't fix this at this point.
I will keep trying and meantime, I downloaded singularity and tried with it, but again, I got an error.
That says,
For input string: ""
-- Check script 'nanopreprocess_s.nf' at line: 172 or see '.nextflow.log' file for more details
Could you help with this?
For your info, with docker "MultiFast5 files detected!" was printed correctly.
yh
Hi,
It might be solved even though this may not be ideal solution.
I struggled with docker, but I realized I have conda env that deeplexicon works.
So I wrote conda path to deeplexicon env into nextflow script.
It is still running, but at least start to process of demultiplexing.
Thank you so much for your help.
If you could find better solutions, I would be happy to try.
yh
Hi, in principle docker - singularity should be the best option but I did not test them in Mac OS. Have you tried singularity?
Dear @yasukoh I tested the pipeline on a Mac OSX. I had to do some small changes since Docker doesn't mount symbolic links. If you do a git pull it should work. Let me know, please
Hi, Thank so much for changes.
I downloaded again and tried. But I got the same error as before.
Is it possibly the problem of only my system??
yh
I don't get it. With Docker you should not see your local environment. So your python environment should be "hidden". Can you please start from scratch and reinstall MoP and try again? The warning you described at the beginning are ok. Let me know.
Hmm. OK, I will uninstall all and re-install everything again.
Give me a bit of time. I will let you know.
Thank you so much for your help.
No, it didn't work.
I got the same error. I didn't copy all of error message before, so the following is the whole message of error.
With that, you might be able to point out something I missed.
Error executing process > 'demultiplexing_with_deeplexicon (deeplexicon-0)'
Caused by:
Process demultiplexing_with_deeplexicon (deeplexicon-0) terminated with an error exit status (137)
Command executed:
ln -s deeplexicon/* .
deeplexicon.py -f multi -p ./ -s 0.9 -m resnet20-final.h5 -b 4000 -v > 0_demux.tsv
Command exit status:
137
Command output:
(empty)
Command error:
2020-08-09 03:03:20.796665: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-08-09 03:03:20.814887: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2304000000 Hz
2020-08-09 03:03:20.816158: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5650c1f1d6f0 executing computations on platform Host. Devices:
2020-08-09 03:03:20.816373: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
/usr/local/lib/python3.7/site-packages/pandas/compat/init.py:117: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
warnings.warn(msg)
Using TensorFlow backend.
info: Verbose mode active - dumping info to stderr
info: DeePlexiCon: 1.0.0
info: arg list: Namespace(batch_size=4000, config=None, form='multi', model='resnet20-final.h5', path='./', segment=None, squiggle=None, threshold=0.9, verbose=True, version=None)
info: Please install GPU version of TF
WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
.command.sh: line 3: 8 Killed deeplexicon.py -f multi -p ./ -s 0.9 -m resnet20-final.h5 -b 4000 -v > 0_demux.tsv
Work dir:
/Users/yh/master_of_pores/NanoPreprocessSimple/work/a4/66129c8d5866b413e9d39310e00854
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh
Does this help?
I greatly appreciate your help.
yh
Ah ok, this looks more a problem with resources. Can you add -profile local and adapt the maximum memory to the one of your system?
The file to be changed is nextflow.global.config
Sorry, I should have copied all of the error log from the beginning.
Could I clarify what should I do?
I modified the part of memory size of local section in nextflow.global.config file to fit to my system.
local {
process {
executor = 'local'
cpus = 1
memory = '8GB' ----- this number
and add to command line -profile local as
"/nextflow run nanopreprocess_s.nf -profile local -with-docker"
By changing these I still get the same error..
What do I still miss??
Definitely my mistake. I am sorry. I forgot to allocate more memory to docker itself.
Once I changed memory allocation of docker, it seems to start demultiplexing.
Thank you so much for your help.
yh
You are welcome!