Tephra docker run issue
mb3188 opened this issue · 9 comments
Hello,
I am trying to run Tephra for the first time using your Docker container. I ran silent run using nohup
in expectation to produce gff file for my assembly
nohup tephra all -c tephra_config.yml
And got following files as output
-rw-r--r-- 1 root root 1462272 Oct 22 20:42 QMD-87_01.1.fasta.index.des
-rw-r--r-- 1 root root 708608 Oct 22 20:42 QMD-87_01.1.fasta.index.md5
-rw-r--r-- 1 root root 167936 Oct 22 20:42 QMD-87_01.1.fasta.index.sds
-rw------- 1 root root 966 Oct 22 20:42 nohup.out
-rw-rw-r-- 1 4886 4886 1764 Oct 20 17:47 tephra_config.yml
-rw------- 1 root root 0 Oct 22 20:42 tephra_suffixerator_errors_oBaE.err
-rw-r--r-- 1 root root 449 Oct 22 20:42 tephra_tair10_full.log
-rw------- 1 root root 12940960 Oct 22 20:42 tephra_transposons_hmmdb_C7Xw.hmm
NO errors in error file,no ssues in log file but the program crashes soon after initiated and nohup.out file contains following message.
"/root/.tephra/gt/bin/gt -j 24 suffixerator -db QMD-87_01.1.fasta -indexname /workdir/TEPHRA/db/QMD-87_01.1.fasta.index -tis -suf -lcp -ssp -sds -des -dna 2> /workdir/TEPHRA/db/tephra_suffixerator_errors_oBaE.err" died to signal "INT" (2) at /usr/local/share/perl/5.26.1/Tephra/Role/Run/GT.pm line 98.
Unable to run 'tephra findltrs'. Here is the exception: "tephra" unexpectedly returned exit value 25 at /usr/local/share/perl/5.26.1/Tephra/Role/Run/TephraCmd.pm line 30.
Not quite sure what is going on. Could you please provide some suggestions of might be happening?
Thanks,
Martina
Hi Martina,
I think this may be a docker issue rather than a tephra problem, but we can figure it out. I would try to run the command again but be sure to put it in the background with & at the end.
nohup tephra all -c tephra_config.yml 2>&1 > tephra.out &
That will send all output to the tephra.out file and put the program in the background. You should be able to then exit the container and re-attach to it later to check on the progress. I'm guessing that without the & the command is running in the foreground and when you exit the process gets killed. That is what I'd like to test first before looking at the data or other possibilities.
Thanks,
Evan
HI Evan,
I have actually ran it in the background before, just did not give you that example above. So that does not seem to be an issue. It starts, produces those files I listed below and then crashes.
The error signal means the process was killed by something from the system or from user input (like CTRL-C). So, if you are running this in the container and the process is getting killed you may be running out of resources.
Can you confirm that you are not running out of disk space? How you create the container will determine where the files are written to. Also, you are requesting 24 CPUs in the command. Can you watch the process to make sure it is not running out of memory or exhausting available resources?
That would be informative at this stage.
Ran now with 60 CPUs and with docker exec, got an error below
docker exec 2342d3d7db1b nohup tephra all -c /workdir/TEPHRA/db/tephra_config.yml --cpu 40 2>&1 > tephra.out &
INFO - Configuration - Log file for monitoring progress and errors: /workdir/TEPHRA/db/tephra_tair10_full.log
INFO - Configuration - Genome file: /workdir/TEPHRA/db/QMD-87_01.1.fasta
INFO - Configuration - Repeat database: /workdir/TEPHRA/db/Human_repeats_library.fa
INFO - Configuration - Number of threads: 100
INFO - Command - 'tephra findltrs' started at: 25-10-2018 03:14:08.
ERROR - 'gt suffixerator' failed with exit value: 135. Here is the output: Bus error (core dumped)
Not sure if that "core dumped" is the memory thing?
Sorry if I was not clear. I would try with fewer threads. Try to set the number to 1 in the configuration file.
Then, see if you can attach and run the tephra command:
docker exec 2342d3d7db1b bash
2342d3d7db1b# cd /workdir/TEPHRA/db/
2342d3d7db1b# nohup tephra all -c tephra_config.yml 2>&1 > tephra.out &
You should be able to watch that output file after that and see if you are getting any errors.
Ok, I ran that as suggested and got two GFF3 files as an output
QMD-87_02.1_ltrdigest85.gff3
QMD-87_02.1_ltrdigest99.gff3
not sure which one is the correct one? I believe the one that is created later, or are both of these intermediate files. In the log file there is never real progress, just that it started but not that it finished or what is doing
Here is the log:
######################
Configuration - Log file for monitoring progress and errors: /workdir/TEPHRA/db/tephra_QMD-87_02.1.log
Configuration - Genome file: /workdir/TEPHRA/db/QMD-87_02.1.fasta.gz
Configuration - Repeat database: /workdir/TEPHRA/db/Human_repeats_library.fa
Configuration - Number of threads: 1
Command - 'tephra findltrs' started at: 26-10-2018 16:31:20.
######################
Also I do not get FASTA of all intact and fragmented transposons in the genome,
and annotation table.
Hi,
The results indicate that the process was killed again. Both of those files are intermediate and should be removed once the 'findltrs' command completes and produces a filtered and refined file of LTR-RTs. I think one problem may be the use of the gzipped genome file. I'm sorry for the inconvenience, but some of the programs used do not allow files to be compressed. I would suggest trying again with an uncompressed genome before moving forward.
I know this is a major inconvenience, but at this stage of development I've focused primarily on the algorithms and biology. You should never have to uncompress anything ideally and I'll add support for this ASAP.
Thanks.
Hi,
Just to update, in the latest version (v0.12.3) compressed input is handled correctly and I've also made changes to the docker image and commands.
Please let me know if there are still issues remaining on this ticket.
Thanks.
I'm going to close this issue as it seems to have been resolved. Please update if you have further issues that are related.