test run hangs on bowtie2-align-s
Closed this issue · 9 comments
Hi,
I'm trying to run the test demo data as specified in the wiki. All software seemed to install fine and I'm not getting any errors in the log file. However, the demo has been running on our server with 32 cores and 256g of ram for four days now and seems to have stalled at the bowtie2-align-s step since the first hour it started running. It is using all the cores but not all the ram. Below is the command I used to run necklace and here is the terminal output
necklace_test_terminal_output.txt
Would any of the logs in .bpipe/ be helpful?
#! /bin/bash
/data3/cherdman/heart_timecourse/ribozero_rnaseq/necklace-1.11/tools/bin/bpipe run \
-n 32 \
-p trinity_options="--max_memory 256G --normalize_reads --no_version_check --trimmomatic" \
-r \
/data3/cherdman/heart_timecourse/ribozero_rnaseq/necklace-1.11/necklace.groovy \
/data3/cherdman/heart_timecourse/ribozero_rnaseq/necklace-1.11/test_sheep/data/data.txt >> \
necklace_test_terminal_output.txt 2>&1
Thank you!
Hi,
I'm not sure what's happened here, but the test data should finish very quickly even with low resources. Can you please try the following and see if the problem persists:
- Stop the current job with "/data3/cherdman/heart_timecourse/ribozero_rnaseq/necklace-1.11/tools/bin/bpipe stop"
- Run again without redirecting the output (remove the >>
necklace_test_terminal_output.txt 2>&1 from the command). If you type "Ctrl-C" while it's running you'll be asked if you want the job to continue and you can type "N" to keep it running in the background. - Use a smaller number of core (e.g. 4) and less memory (e.g. 20G)
Send me the log files in .bpipe/ if it still hangs.
Cheers,
Nadia.
Hi,
Thanks for your quick reply. I started it again 12 hours ago and it still got stuck at bowtie2-align-s. It went through the other steps very quickly and then stayed at the same step for almost the whole time. Again, no error and it didn't output the html file. I've attached the .bpipe folder and here is the command I ran this time:
/data3/cherdman/heart_timecourse/ribozero_rnaseq/necklace-1.11/tools/bin/bpipe run \
-n 4 \
-p trinity_options="--max_memory 20G --normalize_reads --no_version_check --trimmomatic" \
-r \
/data3/cherdman/heart_timecourse/ribozero_rnaseq/necklace-1.11/necklace.groovy \
data/data.txt
Thanks so much,
Chelsea
Hi Chelsea,
It's not clear to me what's wrong here, although I have run into trouble in the past with steps hanging due to incompatible versions of glibc. Can you try running:
/data3/cherdman/heart_timecourse/ribozero_rnaseq/necklace-1.11/tools/bin/bpipe retry test
It should print out the next commands it wants to run (which should be the one that didn't complete previously). Try running the command/commands yourself manually and see if it still hangs. If it completes, you may be able to rerun necklace again with your original command and it will jump to the next step in the pipeline.
Cheers,
Nadia.
Hi Nadia,
Sorry, server crashed last week. I tried rerunning the test using the same command as above and actually got an error this time (see below). It is still running.
* Running CMD: bash -c " set -o pipefail;bowtie2 --local -k 2 --threads 3 -f --score-min G,46,0 -x /data3/cherdman/heart_timecourse/ribozero_rnaseq/necklace-1.11/test_sheep/test_v2/trinity_out_dir/chrysalis/inchworm.K25.L25.DS.fa.min100 /data3/cherdman/heart_timecourse/ribozero_rnaseq/necklace-1.11/test_sheep/test_v2/trinity_out_dir/both.fa | samtools view -@ 3 -F4 -Sb - | samtools sort -m 3579139413 -@ 3 -no - - > /data3/cherdman/heart_timecourse/ribozero_rnaseq/necklace-1.11/test_sheep/test_v2/trinity_out_dir/chrysalis/iworm.bowtie.nameSorted.bam"
Traceback (most recent call last):
File "/usr/local/necklace-1.11/necklace-1.11/tools/bin/hisat2-build", line 95, in <module>
main()
File "/usr/local/necklace-1.11/necklace-1.11/tools/bin/hisat2-build", line 92, in main
os.execv(build_bin_spec, argv)
OSError: [Errno 8] Exec format error
ERROR: Command failed with exit status = 1 :
/usr/local/necklace-1.11/necklace-1.11/tools/bin/hisat2-build data/Ovis_aries.Oar_v3.1.dna.toplevel.chr14.fa genome_guided_assembly/genome
What do you think?
Thanks,
Chelsea
Hi Chelsea,
What happens if you type:
/usr/local/necklace-1.11/necklace-1.11/tools/bin/hisat2-build
do you get usage information?
what about if you type:
/usr/local/necklace-1.11/necklace-1.11/tools/bin/hisat2-build data/Ovis_aries.Oar_v3.1.dna.toplevel.chr14.fa genome_guided_assembly/genome
Cheers,
Nadia.
I get the following:
[cherdman@yost2 necklace-1.11]$ /usr/local/necklace-1.11/necklace-1.11/tools/bin/hisat2-build
Traceback (most recent call last):
File "/usr/local/necklace-1.11/necklace-1.11/tools/bin/hisat2-build", line 95, in
main()
File "/usr/local/necklace-1.11/necklace-1.11/tools/bin/hisat2-build", line 92, in main
os.execv(build_bin_spec, argv)
OSError: [Errno 8] Exec format error
[cherdman@yost2 necklace-1.11]$ /usr/local/necklace-1.11/necklace-1.11/tools/bin/hisat2-build data/Ovis_aries.Oar_v3.1.dna.toplevel.chr14.fa genome_guided_assembly/genome
Traceback (most recent call last):
File "/usr/local/necklace-1.11/necklace-1.11/tools/bin/hisat2-build", line 95, in
main()
File "/usr/local/necklace-1.11/necklace-1.11/tools/bin/hisat2-build", line 92, in main
os.execv(build_bin_spec, argv)
OSError: [Errno 8] Exec format error
Thanks,
Chelsea
Hi Chelsea,
It looks like hisat2 hasn't installed properly. You can download and install it yourself manually, http://daehwankimlab.github.io/hisat2/. You may need to install from source if the binary doesn't work for you. If you get usage information when you run hisat2-build rather than an error you know it's installed okay. Then you can modify the necklace file "tools.groovy" to specify the path to the hisat2 programs.
Cheers,
Nadia.
That was it, thanks! Just ran the test and it completed successfully and quickly.
Excellent! Glad to hear!