Problems with running the pipeline completly

Question

Problems with running the pipeline completly

chalkidiki opened this issue 4 years ago · 18 comments

Dear JAFFA developers,

thanks for providing such a promising tool! I want to use JAFFA on Oxford Nanopore direct cDNA reads, and I already read your recommendations how to run JAFFA with this type of data. However, I am able to run JAFFA on the demo data files correctly, but when I run JAFFA on my cDNA data, it seems to not work completly. The process is not aborted nor I get any kind of error message, but I also don't get any output. I just wanted to ask if this is maybe a problem which is solvable or if it is otherwise possible that JAFFA runs on this kind on data for multiple days (I tried to let it run for about 48 h), and this is the reason why I did not get any output yet.

Below is the output of the commandlog.txt and the console.

Thank you so much

Marius

/media/data/mkle/JAFFA-version-1.09/tools/bin/bpipe run -p readLayout=single -p maxIntron=100 -p minIdTrans=90 -p contigTile=11 -p readTile=11 /media/data/mkle/JAFFA-version-1.09/JAFFA_direct.groovy /media/data/mkle/JAFFA-version-1.09/814_cDNA.fasta
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=./JAVA_TMP

| Starting Pipeline at 2020-06-07 09:47 |

========================================= Stage run_check ==========================================
Running JAFFA version 1.10_dev
Checking for required data files...
/media/data/mkle/JAFFA-version-1.09/hg38_genCode22.fa
/media/data/mkle/JAFFA-version-1.09/hg38_genCode22.tab
/media/data/mkle/JAFFA-version-1.09/known_fusions.txt
/media/data/mkle/JAFFA-version-1.09/hg38.fa
/media/data/mkle/JAFFA-version-1.09/Masked_hg38.1.bt2
/media/data/mkle/JAFFA-version-1.09/hg38_genCode22.1.bt2
All looking good

========================= Stage align_transcripts_to_annotation (814_cDNA) =========================

####################################################################################################

Starting pipeline at Fri Jun 05 15:10:32 CEST 2020

Input files: /media/data/mkle/cDNA/814_cDNA.fasta

Output Log: .bpipe/logs/33748.log

Stage run_check

Stage align_transcripts_to_annotation (814_cDNA)

function run_blat { time /media/data/erst/JAFFA-version-1.09/tools/bin/blat /media/data/mkle/JAFFA-version-1.09/hg38_genCode22.fa /media/data/mkle/cDNA/814_cDNA.fasta -minIdentity=90 -minScore=30 -tileSize=$1 -maxIntron=100 814_cDNA/814_cDNA.psl 2>&1 | tee /media/data/mkle/cDNA/log_blat ; } ; run_blat 11; ### test for the Blat tileSize bug (version 35) ### ; if [[ cat /media/data/mkle/cDNA/log_blat == "Internal error genoFind.c" ]] ; then echo "Blat error with tileSize=11" ; echo "Let's try again with tileSize=15" ; run_blat 15; fi ;
~

Answer 1 · 2020-06-09T07:12:47.000Z

Hi Marius,
Can you please send me the content of the log file .bpipe/logs/33748.log as I'm not sure what the error was. By the way, we happen to be working on a version of JAFFA for ONT data at the moment and I may have a version on github you can clone in the next few weeks if you're happy to wait for that. It's likely to be very developmental and not the final version, but should be much faster and more sensitive than the current release for ONT data.
Cheers,
Nadia.

Answer 2 · 2020-06-09T11:25:07.000Z

Hi Nadia,

You find the output of .bpipe/logs/33748.log below. I am very happy to hear that you currently work on a ONT version of JAFFA. It could be more than useful for a project that we want to publish soon! The last step would be fusion transcript calling on ONT data, so I am very interested in this version.

Best,

Marius

[----] ====================================================================================================
[----] | Starting Pipeline at 2020-06-05 15:10 |
[----] ====================================================================================================
[]
[] ========================================= Stage run_check ==========================================
[814_cDNA]
[814_cDNA] ========================= Stage align_transcripts_to_annotation (814_cDNA) =========================
Exception in thread "Thread-11" java.nio.file.NoSuchFileException: 814_cDNA/.commandlog.txt.swx
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.Files.getLastModifiedTime(Files.java:2266)
at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.invoke(StaticMetaMethodSite.java:46)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:91)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at bpipe.OutputDirectoryWatcher.processEvent(OutputDirectoryWatcher.groovy:139)
at bpipe.OutputDirectoryWatcher$processEvent$1.callCurrent(Unknown Source)
at bpipe.OutputDirectoryWatcher.run(OutputDirectoryWatcher.groovy:122)==[814_cDNA.5] real 2554m48.515s
[814_cDNA.5] user 2551m33.236s Starting Pipeline at 2020-06-05 15:10 [814_cDNA.5] sys 3m22.108s
ERROR: Abnormal termination - check bpipe and operating system has enough memory!~===========================
~]
~~] ========================================= Stage run_check ==============~~===========================
814_cDNA]
814_cDNA] ========================= Stage align_transcripts_to_annotation 814_cDNA) =========================
~~xception in thread "Thread-11" java.nio.file.NoSuchFileException: 814_cDNA/.com~~andlog.txt.swx
~ at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86
~ at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
~ at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
~ at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttribteViews.java:55)
~ at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvidr.java:144)

Answer 3 · 2020-06-10T01:49:21.000Z

Hi Marius,
I'll make sure you are informed about updates to our repository regarding progress with ONT data.
For the error, it almost looks like something has gone amiss with java. What type of system are you running on? How much memory etc. does it have? Have you been able to successfully run the demonstration files through JAFFA to check it is installed correctly and working? (https://github.com/Oshlack/JAFFA/wiki/Example)
Cheers,
Nadia.

Answer 4 · 2020-06-11T11:01:11.000Z

Hi Nadia,

thanks for the updates about the ONT version and for helping me with these issues. Its is of great help for our paper! Regarding system settings: Java is running with 256Mb per thread. Is that enough? Shall I report any additional system settings to you? I am not using any very special sytem, 64 core Ubuntu 16.04.6.

I thought I run the example files correctly, but I double checked and they also produced an error. This error is listed below, also the error message that I get now when I try to run my real data, because it looks however a little bit different now.

Thank you so much!

Marius

Demo files:

Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=./JAVA_TMP
[----] ====================================================================================================
[----] | Starting Pipeline at 2020-06-11 12:38 |
[----] ====================================================================================================
[]
[] ========================================= Stage run_check ==========================================
[BT474-demo_1.fastq]
[BT474-demo_1.fastq] ================================ Stage prepare_reads (BT474-demo_1) ================================
[MCF7-demo_1.fastq]
[MCF7-demo_1.fastq] ================================ Stage prepare_reads (MCF7-demo_1) =================================
[MCF7-demo_2.fastq]
[MCF7-demo_2.fastq] ================================ Stage prepare_reads (MCF7-demo_2) =================================
[BT474-demo_2.fastq]
[BT474-demo_2.fastq] ================================ Stage prepare_reads (BT474-demo_2) ================================
[MCF7-demo_1.fastq]
[MCF7-demo_1.fastq] ================================= Stage get_unmapped (MCF7-demo_1) =================================
[BT474-demo_2.fastq]
[BT474-demo_2.fastq] ================================ Stage get_unmapped (BT474-demo_2) =================================
[BT474-demo_1.fastq]
[BT474-demo_1.fastq] ================================ Stage get_unmapped (BT474-demo_1) =================================
[MCF7-demo_2.fastq]
[MCF7-demo_2.fastq] ================================= Stage get_unmapped (MCF7-demo_2) =================================
[MCF7-demo_1.fastq]
[MCF7-demo_1.fastq] ========================== Stage align_reads_to_annotation (MCF7-demo_1) ===========================
[BT474-demo_1.fastq]
[BT474-demo_1.fastq] ========================== Stage align_reads_to_annotation (BT474-demo_1) ==========================
[BT474-demo_2.fastq]
[BT474-demo_2.fastq] ========================== Stage align_reads_to_annotation (BT474-demo_2) ==========================
[MCF7-demo_2.fastq]
[MCF7-demo_2.fastq] ========================== Stage align_reads_to_annotation (MCF7-demo_2) ===========================
[BT474-demo_1.fastq.12] expr: division by zero
[BT474-demo_2.fastq.10] expr: division by zero
[MCF7-demo_1.fastq.13] expr: division by zero
[MCF7-demo_2.fastq.11] expr: division by zero
ERROR: Command failed with exit status = 2 :

SEQ_COUNT=head -n 1000 MCF7-demo_1.fastq/MCF7-demo_1.fastq.fasta | grep "^>" | wc -l ; SUM_READ_LENGTHS=head -n 1000 MCF7-demo_1.fastq/MCF7-demo_1.fastq.fasta | grep -v ">" | tr -d "\n" | wc --chars ; AVERAGE_READ_LENGTH=expr $SUM_READ_LENGTHS / $SEQ_COUNT ; if [ 0 -eq "0" ] ; then if [ $AVERAGE_READ_LENGTH -le 100 ] ; then readTile=15 ; else readTile=18 ; fi ; else readTile=0 ; fi ; echo "Using tileSize of $readTile" ; function run_blat { time /media/data/erst/JAFFA-version-1.09/tools/bin/blat /media/data/mkle/JAFFA-version-1.09/hg38_genCode22.fa MCF7-demo_1.fastq/MCF7-demo_1.fastq.fasta -minIdentity=98 -minScore=30 -tileSize=$1 -maxIntron=0 MCF7-demo_1.fastq/MCF7-demo_1.fastq.psl 2>&1 | tee /media/data/mkle/JAFFA-version-1.09/MCF7-demo_1.fastq/log_blat ; } ; run_blat $readTile; ### test for the Blat tileSize bug (version 35) ### ; if [[ cat /media/data/mkle/JAFFA-version-1.09/MCF7-demo_1.fastq/log_blat == "Internal error genoFind.c" ]] ; then echo "Blat error with tileSize=$readTile" ; echo "Let's try again with tileSize=15" ; run_blat 15; fi ;

ERROR: Command failed with exit status = 2 :

SEQ_COUNT=head -n 1000 MCF7-demo_2.fastq/MCF7-demo_2.fastq.fasta | grep "^>" | wc -l ; SUM_READ_LENGTHS=head -n 1000 MCF7-demo_2.fastq/MCF7-demo_2.fastq.fasta | grep -v ">" | tr -d "\n" | wc --chars ; AVERAGE_READ_LENGTH=expr $SUM_READ_LENGTHS / $SEQ_COUNT ; if [ 0 -eq "0" ] ; then if [ $AVERAGE_READ_LENGTH -le 100 ] ; then readTile=15 ; else readTile=18 ; fi ; else readTile=0 ; fi ; echo "Using tileSize of $readTile" ; function run_blat { time /media/data/erst/JAFFA-version-1.09/tools/bin/blat /media/data/mkle/JAFFA-version-1.09/hg38_genCode22.fa MCF7-demo_2.fastq/MCF7-demo_2.fastq.fasta -minIdentity=98 -minScore=30 -tileSize=$1 -maxIntron=0 MCF7-demo_2.fastq/MCF7-demo_2.fastq.psl 2>&1 | tee /media/data/mkle/JAFFA-version-1.09/MCF7-demo_2.fastq/log_blat ; } ; run_blat $readTile; ### test for the Blat tileSize bug (version 35) ### ; if [[ cat /media/data/mkle/JAFFA-version-1.09/MCF7-demo_2.fastq/log_blat == "Internal error genoFind.c" ]] ; then echo "Blat error with tileSize=$readTile" ; echo "Let's try again with tileSize=15" ; run_blat 15; fi ;

ERROR: Command failed with exit status = 2 :

SEQ_COUNT=head -n 1000 BT474-demo_1.fastq/BT474-demo_1.fastq.fasta | grep "^>" | wc -l ; SUM_READ_LENGTHS=head -n 1000 BT474-demo_1.fastq/BT474-demo_1.fastq.fasta | grep -v ">" | tr -d "\n" | wc --chars ; AVERAGE_READ_LENGTH=expr $SUM_READ_LENGTHS / $SEQ_COUNT ; if [ 0 -eq "0" ] ; then if [ $AVERAGE_READ_LENGTH -le 100 ] ; then readTile=15 ; else readTile=18 ; fi ; else readTile=0 ; fi ; echo "Using tileSize of $readTile" ; function run_blat { time /media/data/erst/JAFFA-version-1.09/tools/bin/blat /media/data/mkle/JAFFA-version-1.09/hg38_genCode22.fa BT474-demo_1.fastq/BT474-demo_1.fastq.fasta -minIdentity=98 -minScore=30 -tileSize=$1 -maxIntron=0 BT474-demo_1.fastq/BT474-demo_1.fastq.psl 2>&1 | tee /media/data/mkle/JAFFA-version-1.09/BT474-demo_1.fastq/log_blat ; } ; run_blat $readTile; ### test for the Blat tileSize bug (version 35) ### ; if [[ cat /media/data/mkle/JAFFA-version-1.09/BT474-demo_1.fastq/log_blat == "Internal error genoFind.c" ]] ; then echo "Blat error with tileSize=$readTile" ; echo "Let's try again with tileSize=15" ; run_blat 15; fi ;

ERROR: Command failed with exit status = 2 :

SEQ_COUNT=head -n 1000 BT474-demo_2.fastq/BT474-demo_2.fastq.fasta | grep "^>" | wc -l ; SUM_READ_LENGTHS=head -n 1000 BT474-demo_2.fastq/BT474-demo_2.fastq.fasta | grep -v ">" | tr -d "\n" | wc --chars ; AVERAGE_READ_LENGTH=expr $SUM_READ_LENGTHS / $SEQ_COUNT ; if [ 0 -eq "0" ] ; then if [ $AVERAGE_READ_LENGTH -le 100 ] ; then readTile=15 ; else readTile=18 ; fi ; else readTile=0 ; fi ; echo "Using tileSize of $readTile" ; function run_blat { time /media/data/erst/JAFFA-version-1.09/tools/bin/blat /media/data/mkle/JAFFA-version-1.09/hg38_genCode22.fa BT474-demo_2.fastq/BT474-demo_2.fastq.fasta -minIdentity=98 -minScore=30 -tileSize=$1 -maxIntron=0 BT474-demo_2.fastq/BT474-demo_2.fastq.psl 2>&1 | tee /media/data/mkle/JAFFA-version-1.09/BT474-demo_2.fastq/log_blat ; } ; run_blat $readTile; ### test for the Blat tileSize bug (version 35) ### ; if [[ cat /media/data/mkle/JAFFA-version-1.09/BT474-demo_2.fastq/log_blat == "Internal error genoFind.c" ]] ; then echo "Blat error with tileSize=$readTile" ; echo "Let's try again with tileSize=15" ; run_blat 15; fi ;

========================================= Pipeline Failed ==========================================

One or more parallel stages aborted. The following messages were reported:

Branch BT474-demo_1.fastq in stage Unknown reported message:

Command failed with exit status = 2 :

SEQ_COUNT=head -n 1000 BT474-demo_1.fastq/BT474-demo_1.fastq.fasta | grep "^>" | wc -l ; SUM_READ_LENGTHS=head -n 1000 BT474-demo_1.fastq/BT474-demo_1.fastq.fasta | grep -v ">" | tr -d "\n" | wc --chars ; AVERAGE_READ_LENGTH=expr $SUM_READ_LENGTHS / $SEQ_COUNT ; if [ 0 -eq "0" ] ; then if [ $AVERAGE_READ_LENGTH -le 100 ] ; then readTile=15 ; else readTile=18 ; fi ; else readTile=0 ; fi ; echo "Using tileSize of $readTile" ; function run_blat { time /media/data/erst/JAFFA-version-1.09/tools/bin/blat /media/data/mkle/JAFFA-version-1.09/hg38_genCode22.fa BT474-demo_1.fastq/BT474-demo_1.fastq.fasta -minIdentity=98 -minScore=30 -tileSize=$1 -maxIntron=0 BT474-demo_1.fastq/BT474-demo_1.fastq.psl 2>&1 | tee /media/data/mkle/JAFFA-version-1.09/BT474-demo_1.fastq/log_blat ; } ; run_blat $readTile; ### test for the Blat tileSize bug (version 35) ### ; if [[ cat /media/data/mkle/JAFFA-version-1.09/BT474-demo_1.fastq/log_blat == "Internal error genoFind.c" ]] ; then echo "Blat error with tileSize=$readTile" ; echo "Let's try again with tileSize=15" ; run_blat 15; fi ;

Branch BT474-demo_2.fastq in stage Unknown reported message:

Command failed with exit status = 2 :

SEQ_COUNT=head -n 1000 BT474-demo_2.fastq/BT474-demo_2.fastq.fasta | grep "^>" | wc -l ; SUM_READ_LENGTHS=head -n 1000 BT474-demo_2.fastq/BT474-demo_2.fastq.fasta | grep -v ">" | tr -d "\n" | wc --chars ; AVERAGE_READ_LENGTH=expr $SUM_READ_LENGTHS / $SEQ_COUNT ; if [ 0 -eq "0" ] ; then if [ $AVERAGE_READ_LENGTH -le 100 ] ; then readTile=15 ; else readTile=18 ; fi ; else readTile=0 ; fi ; echo "Using tileSize of $readTile" ; function run_blat { time /media/data/erst/JAFFA-version-1.09/tools/bin/blat /media/data/mkle/JAFFA-version-1.09/hg38_genCode22.fa BT474-demo_2.fastq/BT474-demo_2.fastq.fasta -minIdentity=98 -minScore=30 -tileSize=$1 -maxIntron=0 BT474-demo_2.fastq/BT474-demo_2.fastq.psl 2>&1 | tee /media/data/mkle/JAFFA-version-1.09/BT474-demo_2.fastq/log_blat ; } ; run_blat $readTile; ### test for the Blat tileSize bug (version 35) ### ; if [[ cat /media/data/mkle/JAFFA-version-1.09/BT474-demo_2.fastq/log_blat == "Internal error genoFind.c" ]] ; then echo "Blat error with tileSize=$readTile" ; echo "Let's try again with tileSize=15" ; run_blat 15; fi ;

Branch MCF7-demo_1.fastq in stage Unknown reported message:

Command failed with exit status = 2 :

SEQ_COUNT=head -n 1000 MCF7-demo_1.fastq/MCF7-demo_1.fastq.fasta | grep "^>" | wc -l ; SUM_READ_LENGTHS=head -n 1000 MCF7-demo_1.fastq/MCF7-demo_1.fastq.fasta | grep -v ">" | tr -d "\n" | wc --chars ; AVERAGE_READ_LENGTH=expr $SUM_READ_LENGTHS / $SEQ_COUNT ; if [ 0 -eq "0" ] ; then if [ $AVERAGE_READ_LENGTH -le 100 ] ; then readTile=15 ; else readTile=18 ; fi ; else readTile=0 ; fi ; echo "Using tileSize of $readTile" ; function run_blat { time /media/data/erst/JAFFA-version-1.09/tools/bin/blat /media/data/mkle/JAFFA-version-1.09/hg38_genCode22.fa MCF7-demo_1.fastq/MCF7-demo_1.fastq.fasta -minIdentity=98 -minScore=30 -tileSize=$1 -maxIntron=0 MCF7-demo_1.fastq/MCF7-demo_1.fastq.psl 2>&1 | tee /media/data/mkle/JAFFA-version-1.09/MCF7-demo_1.fastq/log_blat ; } ; run_blat $readTile; ### test for the Blat tileSize bug (version 35) ### ; if [[ cat /media/data/mkle/JAFFA-version-1.09/MCF7-demo_1.fastq/log_blat == "Internal error genoFind.c" ]] ; then echo "Blat error with tileSize=$readTile" ; echo "Let's try again with tileSize=15" ; run_blat 15; fi ;

Branch MCF7-demo_2.fastq in stage Unknown reported message:

Command failed with exit status = 2 :

SEQ_COUNT=head -n 1000 MCF7-demo_2.fastq/MCF7-demo_2.fastq.fasta | grep "^>" | wc -l ; SUM_READ_LENGTHS=head -n 1000 MCF7-demo_2.fastq/MCF7-demo_2.fastq.fasta | grep -v ">" | tr -d "\n" | wc --chars ; AVERAGE_READ_LENGTH=expr $SUM_READ_LENGTHS / $SEQ_COUNT ; if [ 0 -eq "0" ] ; then if [ $AVERAGE_READ_LENGTH -le 100 ] ; then readTile=15 ; else readTile=18 ; fi ; else readTile=0 ; fi ; echo "Using tileSize of $readTile" ; function run_blat { time /media/data/erst/JAFFA-version-1.09/tools/bin/blat /media/data/mkle/JAFFA-version-1.09/hg38_genCode22.fa MCF7-demo_2.fastq/MCF7-demo_2.fastq.fasta -minIdentity=98 -minScore=30 -tileSize=$1 -maxIntron=0 MCF7-demo_2.fastq/MCF7-demo_2.fastq.psl 2>&1 | tee /media/data/mkle/JAFFA-version-1.09/MCF7-demo_2.fastq/log_blat ; } ; run_blat $readTile; ### test for the Blat tileSize bug (version 35) ### ; if [[ cat /media/data/mkle/JAFFA-version-1.09/MCF7-demo_2.fastq/log_blat == "Internal error genoFind.c" ]] ; then echo "Blat error with tileSize=$readTile" ; echo "Let's try again with tileSize=15" ; run_blat 15; fi ;

Use 'bpipe errors' to see output from failed commands.

Original File:

[814_cDNA] ========================= Stage align_transcripts_to_annotation (814_cDNA) =========================
Exception in thread "Thread-8" java.nio.file.NoSuchFileException: ./.JAFFA_stages.groovy.swx
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.Files.getLastModifiedTime(Files.java:2266)
at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.invoke(StaticMetaMethodSite.java:46)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:91)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at bpipe.OutputDirectoryWatcher.processEvent(OutputDirectoryWatcher.groovy:139)
at bpipe.OutputDirectoryWatcher$processEvent$1.callCurrent(Unknown Source)
at bpipe.OutputDirectoryWatcher.run(OutputDirectoryWatcher.groovy:122)
[814_cDNA.60] real 5921m31.334s
[814_cDNA.60] user 5914m17.428s
[814_cDNA.60] sys 7m11.320s
ERROR: Abnormal termination - check bpipe and operating system has enough memory!

Answer 5 · 2020-06-24T10:14:34.000Z

Hi Nadia,

do you think this problem is somehow solvable? If you think I should better wait for the ONT version of JAFFA, this is also fine for me. It would be great to get JAFFA running because it could help us a lot with the last piece of work we want to do for an almost finished manuscript.

Best,

Marius

Answer 6 · 2020-06-25T05:28:19.000Z

Hi Marius,

Sorry for the slow reply. Both errors look unfamilar to me. I've just updated the developmental ONT version on github. This is still not optimised, but will do better than version 1 of JAFFA. Do you want to try running it and then we can work through any error that you come across? To install, download https://github.com/Oshlack/JAFFA/archive/fast_jaffa.zip (this is the branch fast_jaffa in the github repository). Install the reference files and run as usual, but use the JAFFA_ONT.groovy mode. ie.
/tools/bin/bpipe run /JAFFA_ONT.groovy /*.fastq.gz

A few notes: The ONT data should be in gzipped fastq format. The other modes of the pipeline (JAFFA_Direct etc.) may not work correctly, so please don't attempt to use this version for short-reads. Due to the high error rate in the data, some reads which support fusions won't align properly to the genome meaning they will be missed or classed as "Low Confidence". You may see many "Low Confidence" calls like this in your data. We're still working on correcting this, but if you think JAFFA's final output is missing some fusions you expect to see, you can also check the files /.txt, which lists fusions candidates flagged earlier in the pipeline (one line per read).

Let me know how you go.

Cheers,
Nadia.

Answer 7 · 2020-06-25T08:21:50.000Z

Hi Nadia,

thanks for sending me the ONT version! I am really excited to get this going! I did everything as you told me and it looks like it works at least better as before (I get many more steps of the pipeline done).

However, another error pops up. Maybe you have an idea how to solve this one?

mkle@laelaps:/media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/.bpipe/logs$ less 37740.log

.bpipe log output:

Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=./JAVA_TMP
[----] ====================================================================================================
[----] | Starting Pipeline at 2020-06-25 10:17 |
[----] ====================================================================================================
[]
[] ========================================= Stage run_check ==========================================
[08KM37_cDNA.fastq]
[08KM37_cDNA.fastq] ================================== Stage get_fasta (08KM37_cDNA) ===================================
[08KM37_cDNA.fastq]
[08KM37_cDNA.fastq] ============================ Stage minimap2_transcriptome (08KM37_cDNA) ============================
[08KM37_cDNA.fastq]
[08KM37_cDNA.fastq] ============================== Stage filter_transcripts (08KM37_cDNA) ==============================
[08KM37_cDNA.fastq.9] bash: line 1: 08KM37_cDNA.fastq/08KM37_cDNA.fastq.paf: Permission denied
ERROR: Command failed with exit status = 1 :

$process_transcriptome_blat_table 08KM37_cDNA.fastq/08KM37_cDNA.fastq.paf 1000 /media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/hg38_genCode22.tab > 08KM37_cDNA.fastq/08KM37_cDNA.fastq.txt

========================================= Pipeline Failed ==========================================

One or more parallel stages aborted. The following messages were reported:

Branch 08KM37_cDNA.fastq in stage Unknown reported message:

Command failed with exit status = 1 :

$process_transcriptome_blat_table 08KM37_cDNA.fastq/08KM37_cDNA.fastq.paf 1000 /media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/hg38_genCode22.tab > 08KM37_cDNA.fastq/08KM37_cDNA.fastq.txt

Use 'bpipe errors' to see output from failed commands.

Cheers,

Marius

Answer 8 · 2020-06-25T08:30:51.000Z

Hi Marius,

Can you check what the variable process_transcriptome_blat_table is equal to in the file, tools.groovy in the place where you installed JAFFA? I suspect that program wasn't installed correctly. Did you receive any errors when you ran the installation script, install_linux64.sh?

Cheers,
Nadia.

Answer 9 · 2020-06-25T09:08:28.000Z

Hi Nadia,

I am sure we can get this done quickly! Thanks so much for helping me! I re-installed everything. During the installation one error pops up:

WARNING: extract_seq_from_fasta could not be found!!!! You will need to download and install extract_seq_from_fasta manually, then add its path to tools.groovy
make_simple_read_table looks like it has been installed
blastn looks like it has been installed
minimap2 looks like it has been installed
process_transcriptome_blat_table looks like it has been installed
bypass_genomic_alignment looks like it has been installed

WARNING: One or more command did not install successfully. See warning messages above. You will need to correct this before running JAFFA.

Do you know where I can get extract_seq_from_fasta from? As far as I can see, process_transcriptome_blat_table looks normal. When I try to run the pipeline now, I can go even further, but another error pops up, maybe connected to the installation problem?

/media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/too
ls/bin/bpipe run /media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA_ONT.groovy /media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/08KM37_cDNA.fastq.gz
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=./JAVA_TMP

| Starting Pipeline at 2020-06-25 11:04 |

========================================= Stage run_check ==========================================

================================== Stage get_fasta (08KM37_cDNA) ===================================

============================ Stage minimap2_transcriptome (08KM37_cDNA) ============================

============================== Stage filter_transcripts (08KM37_cDNA) ==============================

=========================== Stage extract_fusion_sequences (08KM37_cDNA) ===========================
bash: line 1: 08KM37_cDNA.fastq/08KM37_cDNA.fastq.fusions.fa.temp: Permission denied
java -ea -Xmx200m -cp /media/data/erst/JAFFA-version-1.09/tools/bbmap/current/ jgi.ReformatReads in=08KM37_cDNA.fastq/08KM37_cDNA.fastq.fasta out=stdout.fasta fastawrap=0
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=./JAVA_TMP
Executing jgi.ReformatReads [in=08KM37_cDNA.fastq/08KM37_cDNA.fastq.fasta, out=stdout.fasta, fastawrap=0]
Input is being processed as unpaired
Input: 2357866 reads 2614196676 bases
Output: 2357866 reads (100.00%) 2614196676 bases (100.00%)
Time: 12.850 seconds.
Reads Processed: 2357k 183.50k reads/sec
Bases Processed: 2614m 203.45m bases/sec
ERROR: Command failed with exit status = 1 :

cat 08KM37_cDNA.fastq/08KM37_cDNA.fastq.txt | awk '{print $1}' > 08KM37_cDNA.fastq/08KM37_cDNA.fastq.fusions.fa.temp ; /media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/tools/bin/reformat in=08KM37_cDNA.fastq/08KM37_cDNA.fastq.fasta out=stdout.fasta fastawrap=0 | 08KM37_cDNA.fastq/08KM37_cDNA.fastq.fusions.fa.temp > 08KM37_cDNA.fastq/08KM37_cDNA.fastq.fusions.fa ; rm 08KM37_cDNA.fastq/08KM37_cDNA.fastq.fusions.fa.temp ;

========================================= Pipeline Failed ==========================================

One or more parallel stages aborted. The following messages were reported:

Branch 08KM37_cDNA.fastq in stage Unknown reported message:

Command failed with exit status = 1 :

cat 08KM37_cDNA.fastq/08KM37_cDNA.fastq.txt | awk '{print $1}' > 08KM37_cDNA.fastq/08KM37_cDNA.fastq.fusions.fa.temp ; /media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/tools/bin/reformat in=08KM37_cDNA.fastq/08KM37_cDNA.fastq.fasta out=stdout.fasta fastawrap=0 | 08KM37_cDNA.fastq/08KM37_cDNA.fastq.fusions.fa.temp > 08KM37_cDNA.fastq/08KM37_cDNA.fastq.fusions.fa ; rm 08KM37_cDNA.fastq/08KM37_cDNA.fastq.fusions.fa.temp ;

Use 'bpipe errors' to see output from failed commands.

Thank you so much!

Marius

Answer 10 · 2020-06-25T09:11:56.000Z

from tools.groovy:

process_transcriptome_blat_table="/media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/tools/bin/process_transcriptome_blat_table"

extract_seq_from_fasta=""

is empty

Answer 11 · 2020-06-25T10:13:07.000Z

Hi, can you please report the output of
ls -l 08KM37_cDNA.fastq/
I just want to make sure that the "filter_transcripts" stage completed okay.

The missing path to extract_seq_from_fasta is likely to be causing the issue you are getting at the moment. You can try to install this manually with the following commands:
change directory:
cd /media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/tools/
compile the function:
g++ -O3 -o bin/extract_seq_from_fasta ../src/extract_seq_from_fasta.c++
Let me know what the output is and which version of g++ you have.

Answer 12 · 2020-06-25T10:34:07.000Z

Thanks so much for helping me with these issues!

Version:
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.12)

Error when trying g++ -O3 -o bin/extract_seq_from_fasta ../src/extract_seq_from_fas
ta.c++

In file included from /usr/include/c++/5/unordered_set:35:0,
from ../src/extract_seq_from_fasta.c++:19:
/usr/include/c++/5/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
#error This file requires compiler and library support
^
../src/extract_seq_from_fasta.c++: In function ‘int main(int, char**)’:
../src/extract_seq_from_fasta.c++:41:3: error: ‘unordered_set’ was not declared in this scope
unordered_set ids_to_keep;
^
../src/extract_seq_from_fasta.c++:41:23: error: expected primary-expression before ‘>’ token
unordered_set ids_to_keep;
^
../src/extract_seq_from_fasta.c++:41:25: error: ‘ids_to_keep’ was not declared in this scope
unordered_set ids_to_keep;
^

ls -l 08KM37_cDNA.fastq/ output:

-rw-r--r-- 1 mkle pipeline 3153941552 Jun 25 09:57 08KM37_cDNA.fastq.fasta
-rw-r--r-- 1 mkle pipeline 0 Jun 25 11:04 08KM37_cDNA.fastq.fusions.fa
-rw-r--r-- 1 mkle pipeline 0 Jun 25 11:04 08KM37_cDNA.fastq.fusions.fa.temp
-rw-r--r-- 1 mkle pipeline 0 Jun 25 09:57 08KM37_cDNA.fastq.paf
-rw-r--r-- 1 mkle pipeline 0 Jun 25 10:17 08KM37_cDNA.fastq.txt

Answer 13 · 2020-06-25T11:37:18.000Z

No worries. Thanks for your patients with the program.
Can you try this command and let me know if it works:
g++ -std=c++11 -O3 -o bin/extract_seq_from_fasta ../src/extract_seq_from_fas

Unfortunately it looks like the previous steps fails (file sizes are zero). Was the minimap2 path set in tools.groovy?
Could you try removing the minimap2 alignment file:
rm 08KM37_cDNA.fastq/08KM37_cDNA.fastq.paf
and rerunning the pipeline. I suspect you'll still get an error. Which version of Ubuntu are you working with?

Answer 14 · 2020-07-04T18:30:40.000Z

Dear Nadia,

thanks to your suggestions I was able to run JAFFA completly! The results look very good. We have normal RNA-Seq fusion calling and very precise DNA breakpoint data to compare them with, and it already looks like we found some interesting fusion transcripts. This is a great support for our project!

I just have one remaining issue about one of the samples I tried to run. In comparsion with the other samples, this sample however doesn't work and the error I get looks like there is something wrong about the quality of the Nanopore read input, even if there should be no difference to the other samples. This is the error:

/media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/tools/bin/bpipe run -p qin=33 /media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA_ONT.groovy /media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/L16_cDNA.fastq.gz
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=./JAVA_TMP

| Starting Pipeline at 2020-07-04 20:10 |

========================================= Stage run_check ==========================================

==================================== Stage get_fasta (L16_cDNA) ====================================
java -ea -Xmx200m -cp /media/data/erst/JAFFA-version-1.09/tools/bbmap/current/ jgi.ReformatReads in=/media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/L16_cDNA.fastq.gz out=L16_cDNA.fastq/L16_cDNA.fastq.fasta threads=16
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=./JAVA_TMP
Executing jgi.ReformatReads [in=/media/data/mkle/JAFFA-fast_jaffa/JAFFA-fast_jaffa/JAFFA-fast_jaffa/L16_cDNA.fastq.gz, out=L16_cDNA.fastq/L16_cDNA.fastq.fasta, threads=16]
Set threads to 16
Input is being processed as unpaired
Warning! Changed from ASCII-33 to ASCII-64 on input >: 62 -> 31
Up to 13 prior reads may have been generated with incorrect qualities.
If this is a problem you may wish to re-run with the flag 'qin=33' or 'qin=64'.
The ASCII quality encoding offset (64) is not set correctly, or the reads are corrupt; quality value below -5.
Please re-run with the flag 'qin=33' or 'ignorebadquality'.
Problematic read number 13:
@a9aa1031-546d-4d0d-a1af-921278cecaa0 runid=09448d37adae28f9158f3d254f53bc8d4ab2b3e1 read=12 ch=173 start_time=2020-03-20T15:50:42Z flow_cell_id=FAL91727 protocol_group_id=20200320_L-16_cDNA sample_id=L-16
ATCTTCTTTTTTTTTTTTTTTTTTTTTTTTTTTTCGTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTCTAATTACTACCTTTTATTCTAATGTGAACCATGGCCCTGAAAGCTGATAACAAGCTTGTGGCTGAGCAGAAGGGAACTAGGGGTCAGCAGAAAGGATTATGGGTGGAAAACATTAGCTCTTCCTTGGGGAGTGATGCTGGGGAAAGGGAGAGAGTGGCTCAGCCTGCAGGTAAATAGGCTAGGGAAAGAAAGCCCAAGGCCAAAGGCTGGAGGGAGAAGGACAGTCAGCATGTCCAGCCTGGGGTCTGGGTGTAGGGTTATCCCTTCTCCCTGTGCCTTCCCATCTCGTCCATGAGCCTAGGTCTTGGAGCCTTGTGTTGGAGGCTGCTGTGATGTCAGGAACAGGGATCTGTCTAGCTTTGTACTTCCTGGGACCTCCACGCCCCTGTTGACAGATGGAGATTGGGCAGCAGGGCGTGAGGTCCCCAGGAAGTGGCCAAAAGCTAGACAGATCCCCGTTCCTGACATCACAGCAGCCTCCAACACAAGGCTCCAAGACCTAGGCTCATGGACGAGATGGGAAGGCACAGGAGAAGGGATAACCTACACCCAGACCCCAGGCTGGACATGCTGACTTCAATCTCCCCTCCCAGCCTTTGGCCTTGGCTTTTCTAGCCTGCAGGCTAGTCCCTCTCTTCCCTTTCCACAGCATCACTCCCCAGGAAAGAGCCAATGTTTTCCACCCATAATCCTTTCTGCCTCCCCTAGTTCCCTCTGCTCAGCCAAACATTGTCAAGCTTTCGGGGCCATGGTTCACATTAGAATAAAGGTAGTAATTAGAAAAAAAAAAAAAAAAAAAAAAAAACGAAAAAAAAAAAAAAAAAAAAAAGAAGATAGAGCGACAGGCAAGTAGCAATACGTAAC
+
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@A@F@A@@@@@@@A@@@DCHFIFHDLIHCED@FCEGIDBBF@@ihrl@@@@@@@@@@@@gb@@@@@jfcb@@@@@@bcbb@@@@@@@@@@@@@@@@@@@@@@@@@BFFB@C@@j@@@@@@@@@@ag@@acd@BFF@EH@@PB@@@@@de@@A@@@@@@@@@@@adeaa@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@d@@@@EDDEB@B@@@@@@@@@@@@@@b@@fdb@@b@FFBGD@@@@@@@@b@@@@f@@@@@@@@@@@@b@C@@@@@@@@@@f@@@@@@@ACFC@@g@@@@@@@@@b@@@@@@@@@@@A@@badbi@BB@@@@ec@@@@@@@@@@@A@D@@@@@@@@@@@@@b@@@@@@@@@@@@@@@C@@@@@@cd@@@@@d@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@aecd@A@@@@@@@@@@@@@@@@d@@@bc@EGGDBDEE@@@@@@@@@@@e@@@@@@@aaff@CFKEDB@@BEGBBDICIF@@@@@@@@@e@E@FF@@@@@@@@d@@@@@@@@@@@@@@@@b@@HCMDGID@@bdb@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@C@@@chc@@@@@@@@A@@@@@@@@@da@@@@@@@fbb@@@da@@@@@@@@@@@@@@@@@@@@@f@C@DDB@@@@@@@@@@@@@@@efh@@@@@@@@@@@@hgbh@DCA@@@@@@@@@@@@@@@@@@@@@@k@I@@@@@@@@@@@@@@@@@@@@@@Ebiab@C@@@@@@@@@@@@@@@@@@@@@@@@C@@@@@@@@bc@BDEBA@@@@@@@@@
Offset=64
Exception in thread "Thread-1" java.lang.AssertionError
at stream.FASTQ.quadToRead(FASTQ.java:823)
at stream.FASTQ.toReadList(FASTQ.java:705)
at stream.FastqReadInputStream.fillBuffer(FastqReadInputStream.java:111)
at stream.FastqReadInputStream.hasMore(FastqReadInputStream.java:76)
at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(ConcurrentGenericReadInputStream.java:643)
at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGenericReadInputStream.java:635)

Best,

Marius

Answer 15 · 2020-07-07T07:17:33.000Z

Hi Marius,

I'm very happy to hear that JAFFA is finding interesting fusions! I have seen an error similar to this in some data I've been processing, and here is my suggestion:

remove all the files that were generated by JAFFA for your sample, L16_cDNA
find the line:
exec "$reformat in=$input out=$output threads=$threads ;"
in the file JAFFA_ONT.groovy and add either qin=33 or ignorebadquality=t, ie.:
exec "$reformat in=$input out=$output threads=$threads qin=33 ;"
or alternatively:
exec "$reformat in=$input out=$output threads=$threads ignorebadquality=t ;"
Rerun JAFFA as before.

Please let me know if this helps you and if it does, I will add it to the program (quality scores are not currently used anyway).

Cheers,
Nadia.

Answer 16 · 2020-07-07T08:12:32.000Z

Dear Nadia,

it worked! The error did not occur anymore. Thank you so much!

I wanted to ask you also about an error I get when I am running JAFFA direct mode on my short read RNA-Seq data (100bp paired end). For some samples it works, while it doesn't work for others and I am not sure why. The ones where it doesn't work produce no error message, but the Stage_align_reads_to_annotation just takes incredibly long and doesn't seem to really run. Even after two days or so noting changes here, even if the previous steps are rather fast. I attached the .bpipe log file and the command I used below.

Best

Marius

command:

/project/AML-CK/JAFFA/JAFFA-version-1.09//tools/bin/bpipe run /project/AML-CK/JAFFA/JAFFA-version-1.09//JAFFA_direct.groovy /project/AML-CK/data/RNAseq/raw_single/08KM37/*.gz

.bpipe log:

[----] ====================================================================================================
[----] | Starting Pipeline at 2020-07-06 11:10 |
[----] ====================================================================================================
[]
[] ========================================= Stage run_check ==========================================
[8KM37]
[8KM37] =================================== Stage prepare_reads (8KM37) ====================================
[8KM37.68] TrimmomaticPE: Started with arguments: -threads 16 -phred33 /project/AML-CK/data/RNAseq/raw_single/08KM37/8KM37_S38_R_1.fastq.gz /project/AML-CK/data/RNAseq/raw_single/08KM37/8KM37_S38_R_2.fastq.gz /project/AML-CK/JAFFA/JAFFA-version-1.09/8KM37/tempp1.fq /dev/null /project/AML-CK/JAFFA/JAFFA-version-1.09/8KM37/tempp2.fq /dev/null LEADING:0 TRAILING:0 MINLEN:35
[8KM37.68] Input Read Pairs: 60098513 Both Surviving: 60098513 (100.00%) Forward Only Surviving: 0 (0.00%) Reverse Only Surviving: 0 (0.00%) Dropped: 0 (0.00%)
[8KM37.68] TrimmomaticPE: Completed successfully
[8KM37.68] 60098513 reads; of these:
[8KM37.68] 60098513 (100.00%) were paired; of these:
[8KM37.68] 15489369 (25.77%) aligned concordantly 0 times
[8KM37.68] 44609144 (74.23%) aligned concordantly exactly 1 time
[8KM37.68] 0 (0.00%) aligned concordantly >1 times
[8KM37.68] 74.23% overall alignment rate
[8KM37.68] 15489369 reads; of these:
[8KM37.68] 15489369 (100.00%) were paired; of these:
[8KM37.68] 11717820 (75.65%) aligned concordantly 0 times
[8KM37.68] 3771549 (24.35%) aligned concordantly exactly 1 time
[8KM37.68] 0 (0.00%) aligned concordantly >1 times
[8KM37.68] 24.35% overall alignment rate
[8KM37]
[8KM37] ==================================== Stage get_unmapped (8KM37) ====================================
[8KM37.70] 23435640 reads; of these:
[8KM37.70] 23435640 (100.00%) were unpaired; of these:
[8KM37.70] 11695933 (49.91%) aligned 0 times
[8KM37.70] 11739707 (50.09%) aligned exactly 1 time
[8KM37.70] 0 (0.00%) aligned >1 times
[8KM37.70] 50.09% overall alignment rate
[8KM37.70] [bam_sort_core] merging from 6 files...
[8KM37.71] java -ea -Xmx200m -cp /project/AML-CK/JAFFA/JAFFA-version-1.09/tools/bbmap/current/ jgi.ReformatReads in=/project/AML-CK/JAFFA/JAFFA-version-1.09/8KM37/unmapped.fastq out=/project/AML-CK/JAFFA/JAFFA-version-1.09/8KM37/temp.fasta threads=16
[8KM37.71] Executing jgi.ReformatReads [in=/project/AML-CK/JAFFA/JAFFA-version-1.09/8KM37/unmapped.fastq, out=/project/AML-CK/JAFFA/JAFFA-version-1.09/8KM37/temp.fasta, threads=16]
[8KM37.71] Set threads to 16
[8KM37.71] Input is being processed as unpaired
[8KM37.71] Input: 11695933 reads 1169593300 bases
[8KM37.71] Output: 11695933 reads (100.00%) 1169593300 bases (100.00%)
[8KM37.71] Time: 74.311 seconds.
[8KM37.71] Reads Processed: 11695k 157.39k reads/sec
[8KM37.71] Bases Processed: 1169m 15.74m bases/sec
[8KM37.71] java -Djava.library.path=/project/AML-CK/JAFFA/JAFFA-version-1.09/tools/bbmap/jni/ -ea -Xmx812962m -Xms812962m -cp /project/AML-CK/JAFFA/JAFFA-version-1.09/tools/bbmap/current/ jgi.Dedupe sort=d in=/project/AML-CK/JAFFA/JAFFA-version-1.09/8KM37/temp.fasta out=8KM37/8KM37.fasta threads=16 absorbcontainment=f
[8KM37.71] Executing jgi.Dedupe [sort=d, in=/project/AML-CK/JAFFA/JAFFA-version-1.09/8KM37/temp.fasta, out=8KM37/8KM37.fasta, threads=16, absorbcontainment=f]
[8KM37.71] Set threads to 16
[8KM37.71] Initial:
[8KM37.71] Memory: max=816934m, free=799884m, used=17050m
[8KM37.71] Found 2445727 duplicates.
[8KM37.71] Finished exact matches. Time: 20.594 seconds.
[8KM37.71] Memory: max=816934m, free=608534m, used=208400m
[8KM37.71] Input: 11695933 reads 1169593300 bases.
[8KM37.71] Duplicates: 2445727 reads (20.91%) 244572700 bases (20.91%) 0 collisions.
[8KM37.71] Result: 9250206 reads (79.09%) 925020600 bases (79.09%)
[8KM37.71] Sorted output. Time: 27.870 seconds.
[8KM37.71] Memory: max=816934m, free=794497m, used=22437m
[8KM37.71] Printed output. Time: 10.557 seconds.
[8KM37.71] Memory: max=816934m, free=789589m, used=27345m
[8KM37.71] Time: 59.028 seconds.
[8KM37.71] Reads Processed: 11695k 198.14k reads/sec
[8KM37.71] Bases Processed: 1169m 19.81m bases/sec
[8KM37]
[8KM37] ============================= Stage align_reads_to_annotation (8KM37) ==============================
[8KM37.73] Using tileSize of 15

Answer 17 · 2020-07-08T04:01:39.000Z

Unfortunately the original version of JAFFA is extremely slow, and the slowest part of the pipeline does not take multiple threads! This is also something we'd like to fix in a newer version of JAFFA and have been working on.

Your options here are:

Wait longer. It may complete if you leave it for a few more days.
Split the sample into multiple smaller files and run these in parallel. JAFFA can report fusions with 1 read support, so splitting files shouldn't compromise the discovery of most relevant fusions.
Run a different fusion finder such as STAR-fusion or Arriba.

Cheers,
Nadia.

Answer 18 · 2020-07-11T09:29:17.000Z

Hi Nadia,

it took about 5 days but worked in the end. The results also look really good. I will close this issue then. Thanks so much for helping me!

Best,

Marius