An error
Edison2021 opened this issue · 8 comments
Hi there
I ran a trial test using the command line as: princess all --directory analysis --ReadType ccs --ref /home/bwu3/Used_V41_P13_107/p107.ens.fasta --jobs 7 --sampleFiles HiFi.fastq.gz --latency-wait 200 -p -c 2 --verbose
An error popped up below
########
Error in rule mergeAlign:
jobid: 4
output: /mnt/chsrhome/bwu3/test_file/analysis/align/minimap/data.bam
log: /mnt/chsrhome/bwu3/test_file/analysis/align/minimap/merge.log (check log file(s) for error message)
conda-env: /mnt/chsrhome/bwu3/test_file/analysis/.snakemake/conda/a34f2739af6ede4165125922d923477d_
shell:
samtools merge -@ 5 /mnt/chsrhome/bwu3/test_file/analysis/align/minimap/data.bam /mnt/chsrhome/bwu3/test_file/analysis/align/minimap/HiFi.fastq.gz.bam > /mnt/chsrhome/bwu3/test_file/analysis/align/minimap/merge.log 2>&1
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: 3999
Error executing rule mergeAlign on cluster (jobid: 4, external: 3999, jobscript: /mnt/chsrhome/bwu3/test_file/analysis/.snakemake/tmp.a0pkihz4/snakejob.mergeAlign.4.sh). For error details see the cluster log and the log files of the involved rule(s).
########
Could you please give me some suggestions? Thanks
Hi,
Can you please share the error log available here: /mnt/chsrhome/bwu3/test_file/analysis/align/minimap/merge.log
Thanks,
Medhat
The merge.log is empty.
Hmm, do you see a file contains the 3999 prefix from the job cluster_jobid?
It could be in the directory called analysis or in a subdirectory called snake_log
In the file snakejob.mergeAlign.e3999, the message is
bash: xmalloc: .././locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
So, I gave the merging these parameters:
time: "24:00:00"
nCPUs: "8"
mem: 50G
As you can see here https://github.com/MeHelmy/princess/blob/master/cluster/cluster_config.yaml
Princess if the previous configuration is not found it will just adapt what you have. But, from the error, it appears that you do not have enough memory.
How much memory CPU available on your node? (You can check also with the IT administrator.)
Best,
Medhat
Hi Medhat
I disabled the cluster using -e, and it worked for the alignment, merging, and calling SV steps but failed in the calling SNV step.
###############
In the install.sh
wget https://github.com/squeaky-pl/portable-pypy/releases/download/pypy3.6-7.2.0/pypy3.6-7.2.0-linux_x86_64-portable.tar.bz2
mv pypy* pypy.tar.bz2
tar -xvjf pypy.tar.bz2
mv pypy3.* pypy
###############
In the config.yaml
clair_pypy: "/home/source/Clair/pypy3/pypy3.5-7.0.0-linux_x86_64-portable/bin/pypy"
Does the different pypy name cause the calling SNV problem?
Best
Edison
Hi, in the installation script. I install the required pypy and export it. Like here:
cd $clair_dir
wget https://github.com/squeaky-pl/portable-pypy/releases/download/pypy3.6-7.2.0/pypy3.6-7.2.0-linux_x86_64-portable.tar.bz2
mv pypy* pypy.tar.bz2
tar -xvjf pypy.tar.bz2
mv pypy3.* pypy
# tar -xvjf pypy3.6-7.2.0-linux_x86_64-portable.tar.bz2
cd $work_dir
export PATH=$clair_dir/pypy/bin:$PATH
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py
pypy3 -m ensurepip
#pypy3 -m pip install --no-cache-dir intervaltree blosc
pypy3 -m pip install --no-cache-dir intervaltree
pypy3 -m pip install blosc==1.8.3
The clair_pypy: "/home/source/Clair/pypy3/pypy3.5-7.0.0-linux_x86_64-portable/bin/pypy" is absolute, I do not use it anymore
Can you share the error you get?
The error is
Traceback (most recent call last):
File "/mnt/chsrhome/bbww/test_file/analysis/.snakemake/conda/680978645fbe5f728b69ce9ba0c7f486_/bin/clair/../clair.py", line 92, in
main()
File "/mnt/chsrhome/bbww/test_file/analysis/.snakemake/conda/680978645fbe5f728b69ce9ba0c7f486_/bin/clair/../clair.py", line 79, in main
submodule = import_module("%s.%s" % (directory, submodule_name))
File "/mnt/chsrhome/bbww/test_file/analysis/bin/pypy/lib/pypy3.9/importlib/init.py", line 127, in import_module
return bootstrap.gcd_import(name[level:], package, level)
File "", line 1030, in gcd_import
File "", line 1007, in find_and_load
File "", line 986, in find_and_load_unlocked
File "", line 680, in load_unlocked
File "/frozen importlib.bootstrap_external", line 863, in exec_module
File "", line 228, in call_with_frames_removed
File "/mnt/chsrhome/bbww/test_file/analysis/.snakemake/conda/680978645fbe5f728b69ce9ba0c7f486/bin/clair/call_var.py", line 5, in
import numpy as np
ModuleNotFoundError: No module named 'numpy'
[faidx] Truncated sequence: 2:210919337-243193529
[faidx] Truncated sequence: 2:210919337-243193529
Traceback (most recent call last):
File "/mnt/chsrhome/bbww/test_file/analysis/.snakemake/conda/680978645fbe5f728b69ce9ba0c7f486/bin/clair/../clair.py", line 92, in
main()
File "/mnt/chsrhome/bbww/test_file/analysis/.snakemake/conda/680978645fbe5f728b69ce9ba0c7f486/bin/clair/../clair.py", line 86, in main
submodule.main()
File "/mnt/chsrhome/bbww/test_file/analysis/.snakemake/conda/680978645fbe5f728b69ce9ba0c7f486/bin/dataPrepScripts/CreateTensor.py", line 440, in main
OutputAlnTensor(args)
File "/mnt/chsrhome/bbww/test_file/analysis/.snakemake/conda/680978645fbe5f728b69ce9ba0c7f486/bin/dataPrepScripts/CreateTensor.py", line 372, in OutputAlnTensor
tensor_fp.stdin.write(l)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/mnt/chsrhome/bbww/test_file/analysis/.snakemake/conda/680978645fbe5f728b69ce9ba0c7f486/bin/clair/../clair.py", line 92, in
main()
File "/mnt/chsrhome/bbww/test_file/analysis/.snakemake/conda/680978645fbe5f728b69ce9ba0c7f486/bin/clair/../clair.py", line 86, in main
submodule.main()
File "/mnt/chsrhome/bbww/test_file/analysis/.snakemake/conda/680978645fbe5f728b69ce9ba0c7f486/bin/dataPrepScripts/ExtractVariantCandidates.py", line 450, in main
make_candidates(args)
File "/mnt/chsrhome/bbww/test_file/analysis/.snakemake/conda/680978645fbe5f728b69ce9ba0c7f486_/bin/dataPrepScripts/ExtractVariantCandidates.py", line 367, in make_candidates
can_fp.stdin.write(output)
BrokenPipeError: [Errno 32] Broken pipe
By the way, I have installed NumPy
Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
import numpy
numpy.version.version
'1.21.5'