errors during the run of Augustus

Question

errors during the run of Augustus

Closed this issue 5 years ago · 43 comments

Hi Luigi!

I ran the pipeline (options: --stranded --proteins --short_reads --adapter --mask_genome --max_intron_length 10000 ) and it created an error:
Traceback (most recent call last):

File "/opt/LoReAn/code/lorean.py", line 560, in
main()
File "/opt/LoReAn/code/lorean.py", line 288, in main
augustus_file, genemark_file = inputEvm.braker_folder_find(braker_folder)
File "/opt/LoReAn/code/prepareEvmInputs.py", line 115, in braker_folder_find
gff = [y for x in os.walk(location) for y in glob(os.path.join(x[0], "augustus.hints.gff"))][0]
IndexError: list index out of range

Any suggestions?

Unfortunately, I cannot provide much more details of the run since all intermediate files (outputs of previous tools) were deleted with this crash...

Is it possible to include intermediates in future versions and restart from where the pipeline crashed?!

Thanks for your help
Michael

Answer 1 · 2019-07-22T11:10:58.000Z

I think that I solved it. I need few days to make a new image.

Answer 2 · 2019-07-22T12:06:03.000Z

That's awesome! Thanks! Looking forward to try again...
Michael

Answer 3 · 2019-07-29T11:42:06.000Z

Hi Luigi!
Any updates on this?
Best
Michael

Answer 4 · 2019-07-30T11:54:59.000Z

@michieitel
i made a new image. Can you test it?

cheers
Luigi

Answer 5 · 2019-07-30T13:01:25.000Z

Hi Luigi!

Thanks for creating a new image. How can I check it is the latest?

I pulled with:
singularity pull docker://lfaino/lorean:noIPRS

when running 'lorean -h' it says 2017 in the last line...!?

cheers
Michael

Answer 6 · 2019-07-30T13:11:11.000Z

@michieitel
please work with

singularity pull docker://lfaino/lorean:latest

cheers

Answer 7 · 2019-07-30T13:25:23.000Z

ah I see. thanks. will try and report

Answer 8 · 2019-07-30T13:41:31.000Z

I pulled using the link above... still says 2017 ;-)

But since I got the lorean_latest.sif I assume it should be the correct version...

Answer 9 · 2019-07-30T13:50:00.000Z

i guess so. I never noticed

Answer 10 · 2019-07-30T17:13:42.000Z

same error again:

Traceback (most recent call last):
File "/opt/LoReAn/code/lorean.py", line 596, in
main()
File "/opt/LoReAn/code/lorean.py", line 320, in main
augustus_file, genemark_file = inputEvm.braker_folder_find(braker_folder)
File "/opt/LoReAn/code/prepareEvmInputs.py", line 115, in braker_folder_find
gff = [y for x in os.walk(location) for y in glob(os.path.join(x[0], "augustus.hints.gff"))][0]
IndexError: list index out of range

PLUS all intermediated removed automatically again...

Answer 11 · 2019-07-30T17:15:44.000Z

@michieitel
can you send me the singularity command?
thanks

Answer 12 · 2019-07-30T17:20:32.000Z

singularity exec \
-B /home/ubuntu/cbas/lorean/config/:/opt/LoReAn/third_party/software/augustus/config/ \
-B /home/ubuntu/cbas/lorean/Libraries/:/usr/local/RepeatMasker/Libraries/ \
/home/ubuntu/cbas/lorean/lorean_latest.sif \
lorean -t 20 -sp cbas_masurca2 \
--stranded \
--proteins /home/ubuntu/cbas/lorean/data/PORI_Demo_Amphimedon_queenslandica_v2.1__P__FERNANDEZ-VALVERDE.fasta \
--short_reads /home/ubuntu/cbas/lorean/data/CBAS_Concatenated_RNAseq_Read1_Clean_Datasets.fastq,/home/ubuntu/cbas/lorean/data/CBAS_Concatenated_RNAseq_Read2_Clean_Datasets.fastq \
--long_reads /home/ubuntu/cbas/lorean/data/cbas_cDNA_polyA-guppy-3.1.5-hac.porechop_100bp_to_20kb_combined.fastq \
--adapter /home/ubuntu/cbas/lorean/data/TruSeq3-PE-2.fa \
--mask_genome \
--working_dir lorean_run1 \
--max_intron_length 40000 \
/home/ubuntu/cbas/lorean/data/CBAS_MASURCA-2_final.genome.scf.fasta 2> cbas_lorean_run1.log

Answer 13 · 2019-07-30T17:22:12.000Z

we can try adding add --keep_tmp to lorean options. The folder will not disappear.
do you have the geneMark key in the home folder of the user that run LoReAn?

Answer 14 · 2019-07-30T17:23:02.000Z

ubuntu@lorean:~/cbas/lorean$ cat ~/.gm_key
TTGTTCAATTAGCACGGATGTTTTTTTTTTTTTTTTCCGTCGCCATAAAGTTACTAACAGAATTCAAAAGGGAGCGCATA
520951310

Answer 15 · 2019-07-30T17:24:03.000Z

If it helps to keep inmtermediates to find the error I can run again with the suggested option ...

Answer 16 · 2019-07-30T17:25:28.000Z

ok
I will test the image again to see if something is wrong.

Answer 17 · 2019-07-30T17:26:00.000Z

should I run it on a dummy set?

Answer 18 · 2019-07-30T17:26:56.000Z

can you include a test dataset in the repo that we can both run?

Answer 19 · 2019-07-30T17:28:04.000Z

it is there.

https://github.com/lfaino/LoReAn_Example

Answer 20 · 2019-07-30T17:28:05.000Z

oh there is one. I will try using that one!

Answer 21 · 2019-07-30T17:55:32.000Z

not sure if I can ask here, but is it too difficult to implement an option that allows to pass user-defined options for the gmap step of long reads.... I am asking since I have figured out the settings that worked best for my set of nanopore reads.

Answer 22 · 2019-07-30T17:59:16.000Z

which setting do you mean? can you tell me the option?

Answer 23 · 2019-07-30T18:01:46.000Z

on the toy dataset all works fine...
is /home/ubuntu/cbas/lorean/config/ folder writing accessible?

Answer 24 · 2019-07-30T18:36:59.000Z

for gmap settings I am using:
-k 15 -B 4 --cross-species -A --exons=cdna --format=samse --npaths=0 --sam-extended-cigar

not sure which of these you included...

Answer 25 · 2019-07-30T18:37:19.000Z

home/ubuntu/cbas/lorean/config/ is accessible

Answer 26 · 2019-07-30T18:38:02.000Z

I got the same error for both example datasets ...

Answer 27 · 2019-07-30T18:39:06.000Z

example 1: Plicaturopsis

singularity exec \
-B /home/ubuntu/cbas/lorean/config/:/opt/LoReAn/third_party/software/augustus/config/ \
-B /home/ubuntu/cbas/lorean/Libraries/:/usr/local/RepeatMasker/Libraries/ \
/home/ubuntu/cbas/lorean/lorean_latest.sif \
lorean -a -d -f -mg -t 20 --keep_tmp -rp repeats.scaffold3.bed \
-sr /home/ubuntu/cbas/lorean/LoReAn_Example/Crispa/scaffold3.short_1.fastq,/home/ubuntu/cbas/lorean/LoReAn_Example/Crispa/scaffold3.short_2.fastq \
-lr /home/ubuntu/cbas/lorean/LoReAn_Example/Crispa/scaffold3.long.fasta \
-pr /home/ubuntu/cbas/lorean/LoReAn_Example/Crispa/scaffold3.prot.fasta \
-sp crispa \
--working_dir Plicaturopsis \
/home/ubuntu/cbas/lorean/LoReAn_Example/Crispa/scaffold3.fasta

example 2: Verticillium

singularity exec \
-B /home/ubuntu/cbas/lorean/config/:/opt/LoReAn/third_party/software/augustus/config/ \
-B /home/ubuntu/cbas/lorean/Libraries/:/usr/local/RepeatMasker/Libraries/ \
/home/ubuntu/cbas/lorean/lorean_latest.sif \
lorean -t 20 --keep_tmp -a -f -d \
-sr /home/ubuntu/cbas/lorean/LoReAn_Example/JR2/readsChr.subset.fastq \
-lr /home/ubuntu/cbas/lorean/LoReAn_Example/JR2/longReadsChr8.fastq \
-pr /home/ubuntu/cbas/lorean/LoReAn_Example/JR2/subset.prot.fasta \
-sp JR2 \
--working_dir Verticillium \
-mg /home/ubuntu/cbas/lorean/LoReAn_Example/JR2/chr8.fasta

Answer 28 · 2019-07-30T18:41:45.000Z

In the beaker folder in run folder, you should see a genemark folder.

Can you see any error or log file?

Can you check if you get any errors?

Answer 29 · 2019-07-30T18:42:16.000Z

what I did not understand for the examples was you specify adaptors with '-a' but then don't pürovide a file? what is the default?

Answer 30 · 2019-07-30T18:42:53.000Z

There is a module that looks for them

Answer 31 · 2019-07-30T18:43:52.000Z

so I don't have to specify the adaptor file? it is included in the image?

Answer 32 · 2019-07-30T18:44:15.000Z

how can I access the braker folder of the image?

Answer 33 · 2019-07-30T18:45:24.000Z

It is better if you specify but without it will work.

Can you find out if genemark worked in the braker folder?

Answer 34 · 2019-07-30T18:45:56.000Z

how can I access the braker folder of the image?

Answer 35 · 2019-07-30T18:47:05.000Z

It is not in the image. In the folder where you run there is a LoReAn folder. Can you see it? Ottieni Outlook per Android<https://aka.ms/ghei36>

…

________________________________ From: michieitel <notifications@github.com> Sent: Tuesday, July 30, 2019 8:45:56 PM To: lfaino/LoReAn <LoReAn@noreply.github.com> Cc: Luigi Faino <luigi.faino@uniroma1.it>; Comment <comment@noreply.github.com> Subject: Re: [lfaino/LoReAn] errors during the run of Augustus (#26) how can I access the braker folder of the image? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#26?email_source=notifications&email_token=AA45GFDS7HKK2P7Z4DNCFMDQCCD6JA5CNFSM4IFXBYI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3E5WJQ#issuecomment-516545318>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AA45GFBZO2GN4ZFK6I7ZJ73QCCD6JANCNFSM4IFXBYIQ>.

--

___________________________________________ *Il tuo 5 diventa 1000* Fai crescere la tua università Dona il 5 per mille alla Sapienza Codice fiscale: *80209930587* https://www.uniroma1.it/it/pagina/fai-crescere-la-tua-universita-con-il-cinque-mille <https://www.uniroma1.it/it/pagina/fai-crescere-la-tua-universita-con-il-cinque-mille>

Answer 36 · 2019-07-30T18:49:18.000Z

error log for example 1:

Use of uninitialized value $epath in concatenation (.) or string at /opt/LoReAn/third_party/software/BRAKER/scripts//braker.pl line 2370.
ERROR in file /opt/LoReAn/third_party/software/BRAKER/scripts//braker.pl at line 5616
Failed to execute: perl /opt/LoReAn/third_party/software/gm_et_linux_64/gmes_petap//gmes_petap.pl --verbose --sequence=/home/ubuntu/cbas/lorean/LoReAn_Plicaturopsis/run/braker/genome.fa --ET=/home/ubuntu/cbas/lorean/LoReAn_Plicaturopsis/run/braker/genemark_hintsfile.gff --et_score 10 --max_intergenic 50000 --cores=9 --fungus 1>/home/ubuntu/cbas/lorean/LoReAn_Plicaturopsis/run/braker/GeneMark-ET.stdout 2>/home/ubuntu/cbas/lorean/LoReAn_Plicaturopsis/run/braker/errors/GeneMark-ET.stderr

looks like genemark cannot be fired up? key-related?

Answer 37 · 2019-07-30T18:51:46.000Z

Can you check the .error file of genemark?

What is inside?

Answer 38 · 2019-07-30T18:54:43.000Z

no .err in GenMark folder. content:

drwxr-xr-x 6 ubuntu ubuntu 4.0K Jul 30 17:55 GeneMark-ET
-rw-r--r-- 1 ubuntu ubuntu 1.5K Jul 30 17:55 GeneMark-ET.stdout
-rw-r--r-- 1 ubuntu ubuntu 10 Jul 30 17:54 bam_header.map
-rw-r--r-- 1 ubuntu ubuntu 714 Jul 30 17:55 braker.error.log
-rw-r--r-- 1 ubuntu ubuntu 9.0K Jul 30 17:54 braker.log
drwxr-xr-x 2 ubuntu ubuntu 4.0K Jul 30 17:54 errors
-rw-r--r-- 1 ubuntu ubuntu 436K Jul 30 17:54 genemark_hintsfile.gff
-rw-r--r-- 1 ubuntu ubuntu 2.4M Jul 30 17:54 genome.fa
-rw-r--r-- 1 ubuntu ubuntu 10 Jul 30 17:54 genome_header.map
-rw-r--r-- 1 ubuntu ubuntu 436K Jul 30 17:54 hintsfile.gff
drwxr-xr-x 2 ubuntu ubuntu 4.0K Jul 30 17:54 species

Answer 39 · 2019-07-30T18:55:57.000Z

This

/home/ubuntu/cbas/lorean/LoReAn_Plicaturopsis/run/braker/errors/GeneMark-ET.stderr

Answer 40 · 2019-07-30T18:56:42.000Z

GeneMark.hmm eukaryotic 3
GeneMark.hmm eukaryotic 3
Your license period has ended. We hope that you found this
Your license period has ended. We hope that you found this
software useful. If you would like to renew this license,
software useful. If you would like to renew this license,
please contact GeneProbe, Inc. at custserv@genepro.com
please contact GeneProbe, Inc. at custserv@genepro.com

(in cleanup) Can't call method "FETCH" on an undefined value at /usr/local/share/perl/5.22.1/Object/InsideOut.pm line 1953 during global destruction.
(in cleanup) Can't call method "FETCH" on an undefined value at /usr/local/share/perl/5.22.1/Object/InsideOut.pm line 1953 during global destruction.
(in cleanup) Can't call method "FETCH" on an undefined value at /usr/local/share/perl/5.22.1/Object/InsideOut.pm line 1953 during global destruction.

Answer 41 · 2019-07-30T18:58:11.000Z

The key of genemark is expired. You need a new one

Answer 42 · 2019-07-30T19:00:10.000Z

I feel stupid now... let me get it and run again.
many thanks for now

Answer 43 · 2019-08-05T09:29:26.000Z

Hi Luigi,

the pipeline finished. It was that error. Now playing with weighings of the datasets.

Many Thanks for your help!
Michael