Illumina/manta

generateCandidateSV taskExitCode -11

bryce-turner opened this issue · 2 comments

bryce-turner commented

I've encountered an issue that I haven't been quite able to figure out. This is a common issue among some recent data that results in a Failed to complete command task: 'generateCandidateSV_0000' launched from master workflow when running on whole genomes in somatic calling mode. I've initially tested across all chromosomes in reference, then supplied a bed with only the primary chromosomes (1-22, X, Y). Still failed with the same error. Now I've tried running on each chromosome, and the resulting error only appears on chromosome 2, at least in this case.

I'm curious to know if there are some potential options that need to be changed. This is running on ~120x constitutional with ~40x tumor.

Here is the snippet of the error:

    [2020-02-10T18:33:18.222349Z] [dback-c12-n01.tgen.org] [33455_1] [TaskManager] [ERROR] [generateCandidateSV_0000] Error Message:
    [2020-02-10T18:33:18.223755Z] [dback-c12-n01.tgen.org] [33455_1] [TaskManager] [ERROR] [generateCandidateSV_0000] Anomalous task wrapper stderr output. Wrapper signal file: '/scratch/tgenjetstream/centro/projects/MMRF_2465_24cd84859d/temp/genome/somatic_structural_calls/manta/MMRF_2465_2_PB_WBC_C3_KHWGS-MMRF_2465_2_BM_CD138pos_T1_KHWGS_2/workspace/pyflow.data/logs/tmp/taskWrapperLogs/000/040/pyflowTaskWrapper.signal.txt'
    [2020-02-10T18:33:18.225209Z] [dback-c12-n01.tgen.org] [33455_1] [TaskManager] [ERROR] [generateCandidateSV_0000] Logging 5 line(s) of task wrapper log output below:
    [2020-02-10T18:33:18.227151Z] [dback-c12-n01.tgen.org] [33455_1] [TaskManager] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-02-10T18:29:01.154569Z] [dback-c12-n01.tgen.org] [33455_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] wrapperStart
    [2020-02-10T18:33:18.228381Z] [dback-c12-n01.tgen.org] [33455_1] [TaskManager] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-02-10T18:29:01.166108Z] [dback-c12-n01.tgen.org] [33455_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] taskStart
    [2020-02-10T18:33:18.229558Z] [dback-c12-n01.tgen.org] [33455_1] [TaskManager] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-02-10T18:33:18.192996Z] [dback-c12-n01.tgen.org] [33455_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] taskExitCode -11
    [2020-02-10T18:33:18.230691Z] [dback-c12-n01.tgen.org] [33455_1] [TaskManager] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-02-10T18:33:18.197044Z] [dback-c12-n01.tgen.org] [33455_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] taskStderrTail 1
    [2020-02-10T18:33:18.232387Z] [dback-c12-n01.tgen.org] [33455_1] [TaskManager] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] Last 0 stderr lines from task (of 0 total lines):
    [2020-02-10T18:33:18.233556Z] [dback-c12-n01.tgen.org] [33455_1] [TaskManager] [ERROR] Shutting down task submission. Waiting for remaining tasks to complete.
    [2020-02-10T18:33:26.100225Z] [dback-c12-n01.tgen.org] [33455_1] [WorkflowRunner] [ERROR] Workflow terminated due to the following task errors:
    [2020-02-10T18:33:26.102834Z] [dback-c12-n01.tgen.org] [33455_1] [WorkflowRunner] [ERROR] Failed to complete command task: 'generateCandidateSV_0000' launched from master workflow, error code: 1, command: '/packages/easy-build/software/manta/1.6.0/libexec/GenerateSVCandidates --threads 19 --align-stats /scratch/tgenjetstream/centro/projects/MMRF_2465_24cd84859d/temp/genome/somatic_structural_calls/manta/MMRF_2465_2_PB_WBC_C3_KHWGS-MMRF_2465_2_BM_CD138pos_T1_KHWGS_2/workspace/alignmentStats.xml --graph-file /scratch/tgenjetstream/centro/projects/MMRF_2465_24cd84859d/temp/genome/somatic_structural_calls/manta/MMRF_2465_2_PB_WBC_C3_KHWGS-MMRF_2465_2_BM_CD138pos_T1_KHWGS_2/workspace/svLocusGraph.bin --bin-index 0 --bin-count 1 --max-edge-count 10 --min-candidate-sv-size 8 --min-candidate-spanning-count 3 --min-scored-sv-size 50 --ref /home/tgenref/homo_sapiens/grch38_hg38/hg38tgen/genome_reference/GRCh38tgen_decoy_alts_hla.fa --candidate-output-file /scratch/tgenjetstream/centro/projects/MMRF_2465_24cd84859d/temp/genome/somatic_structural_calls/manta/MMRF_2465_2_PB_WBC_C3_KHWGS-MMRF_2465_2_BM_CD138pos_T1_KHWGS_2/workspace/svHyGen/candidateSV.0000.vcf --diploid-output-file /scratch/tgenjetstream/centro/projects/MMRF_2465_24cd84859d/temp/genome/somatic_structural_calls/manta/MMRF_2465_2_PB_WBC_C3_KHWGS-MMRF_2465_2_BM_CD138pos_T1_KHWGS_2/workspace/svHyGen/diploidSV.0000.vcf --min-qual-score 10 --min-pass-qual-score 20 --min-pass-gt-score 15 --somatic-output-file /scratch/tgenjetstream/centro/projects/MMRF_2465_24cd84859d/temp/genome/somatic_structural_calls/manta/MMRF_2465_2_PB_WBC_C3_KHWGS-MMRF_2465_2_BM_CD138pos_T1_KHWGS_2/workspace/svHyGen/somaticSV.0000.vcf --min-somatic-score 10 --min-pass-somatic-score 30 --enable-remote-read-retrieval --chrom-depth /scratch/tgenjetstream/centro/projects/MMRF_2465_24cd84859d/temp/genome/somatic_structural_calls/manta/MMRF_2465_2_PB_WBC_C3_KHWGS-MMRF_2465_2_BM_CD138pos_T1_KHWGS_2/workspace/chromDepth.txt --edge-runtime-log /scratch/tgenjetstream/centro/projects/MMRF_2465_24cd84859d/temp/genome/somatic_structural_calls/manta/MMRF_2465_2_PB_WBC_C3_KHWGS-MMRF_2465_2_BM_CD138pos_T1_KHWGS_2/workspace/svHyGen/edgeRuntimeLog.0000.txt --edge-stats-log /scratch/tgenjetstream/centro/projects/MMRF_2465_24cd84859d/temp/genome/somatic_structural_calls/manta/MMRF_2465_2_PB_WBC_C3_KHWGS-MMRF_2465_2_BM_CD138pos_T1_KHWGS_2/results/stats/svCandidateGenerationStats.xml --edge-stats-report /scratch/tgenjetstream/centro/projects/MMRF_2465_24cd84859d/temp/genome/somatic_structural_calls/manta/MMRF_2465_2_PB_WBC_C3_KHWGS-MMRF_2465_2_BM_CD138pos_T1_KHWGS_2/results/stats/svCandidateGenerationStats.tsv --align-file /scratch/tgenjetstream/centro/projects/MMRF_2465_24cd84859d/genome/alignment/bwa/MMRF_2465_2_PB_WBC_C3_KHWGS/MMRF_2465_2_PB_WBC_C3_KHWGS.bwa.bam --tumor-align-file /scratch/tgenjetstream/centro/projects/MMRF_2465_24cd84859d/genome/alignment/bwa/MMRF_2465_2_BM_CD138pos_T1_KHWGS/MMRF_2465_2_BM_CD138pos_T1_KHWGS.bwa.bam'
    [2020-02-10T18:33:26.104167Z] [dback-c12-n01.tgen.org] [33455_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] Error Message:
    [2020-02-10T18:33:26.105409Z] [dback-c12-n01.tgen.org] [33455_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] Anomalous task wrapper stderr output. Wrapper signal file: '/scratch/tgenjetstream/centro/projects/MMRF_2465_24cd84859d/temp/genome/somatic_structural_calls/manta/MMRF_2465_2_PB_WBC_C3_KHWGS-MMRF_2465_2_BM_CD138pos_T1_KHWGS_2/workspace/pyflow.data/logs/tmp/taskWrapperLogs/000/040/pyflowTaskWrapper.signal.txt'
    [2020-02-10T18:33:26.106629Z] [dback-c12-n01.tgen.org] [33455_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] Logging 5 line(s) of task wrapper log output below:
    [2020-02-10T18:33:26.107776Z] [dback-c12-n01.tgen.org] [33455_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-02-10T18:29:01.154569Z] [dback-c12-n01.tgen.org] [33455_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] wrapperStart
    [2020-02-10T18:33:26.109008Z] [dback-c12-n01.tgen.org] [33455_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-02-10T18:29:01.166108Z] [dback-c12-n01.tgen.org] [33455_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] taskStart
    [2020-02-10T18:33:26.110242Z] [dback-c12-n01.tgen.org] [33455_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-02-10T18:33:18.192996Z] [dback-c12-n01.tgen.org] [33455_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] taskExitCode -11
    [2020-02-10T18:33:26.111508Z] [dback-c12-n01.tgen.org] [33455_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-02-10T18:33:18.197044Z] [dback-c12-n01.tgen.org] [33455_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] taskStderrTail 1
    [2020-02-10T18:33:26.112631Z] [dback-c12-n01.tgen.org] [33455_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] Last 0 stderr lines from task (of 0 total lines):

Please let me know if there is any other information that would be helpful! I've also attached the full log in hopes that that is more helpful.
manta_MMRF_2465_2_PB_WBC_C3_KHWGS-MMRF_2465_1_BM_CD138pos_T1_KHWGL_bwa_2.log

bryce-turner commented

Some additional notes on this. I was able to run both the constitutional/normal bam and tumor bam by themselves through manta without any issues. It's the somatic analysis that has issues.

bryce-turner commented

Another update, as it might help other users that run into a similar problem. The version of manta that is being used is one that was built using EasyBuild. However, when using the precompiled binaries/libraries provided in the release we no longer run into the issue mentioned above. Therefore it is evident that this is a compilation issue, not a true bug to manta.

I'm closing this as a result of finding this solution.