variable declaration prevents workflow from running in parallel in dxCompiler 2.10.2
Closed this issue · 5 comments
Hi dxCompiler team,
One of the scientists I work with is reporting an issue with one of her workflows in that an execution is not starting on DNAnexus even though all of its inputs are satisfied.
Here is the basic workflow definition. When run on DNAnexus, it appears that the tasks mutect2
strelka
and the scatter
block all execute concurrently, however muse
does not. It waits for the scatter block to be complete before executing, even though none of its inputs require anything from the block.
Moving muse
up in the WDL does not resolve the issue.
version 1.0
import "bam_tools.wdl" as bam_tools
import "vcf_tools.wdl" as vcf_tools
import "mutect2.wdl" as mutect2
import "muse.wdl" as muse
workflow call_snps {
input {
File tumor_bam
File normal_bam
File genome_tarball
File intervals
File known_sites
File known_sites_index
}
call mutect2.mutect2 {
input:
tumor_bam = tumor_bam,
normal_bam = normal_bam,
genome_tarball = genome_tarball,
intervals = intervals
}
call strelka {
input:
tumor_bam = tumor_bam,
normal_bam = normal_bam,
genome_tarball = genome_tarball
}
Array[String] chr_num = ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "X", "Y", "M"]
Array[String] chrs = prefix("chr", chr_num)
scatter (chr in chrs) {
call bam_tools.split_bam as split_tumor_bam {
input:
chr = chr,
bam = tumor_bam,
}
call bam_tools.split_bam as split_normal_bam {
input:
chr = chr,
bam = normal_bam,
}
call lofreq {
input:
tumor_bam = split_tumor_bam.chr_bam,
normal_bam = split_normal_bam.chr_bam,
genome_tarball = genome_tarball,
chr = chr,
known_sites = known_sites,
known_sites_index = known_sites_index,
}
}
call muse.muse {
input:
tumor_bam = tumor_bam,
normal_bam = normal_bam,
genome_tarball = genome_tarball,
chrs = chrs,
known_sites = known_sites,
}
call vcf_tools.bcftools_concat_vcfs as merge_lofreq {
input:
vcfs = select_all(lofreq.somatic_vcf),
caller_name = "lofreq"
}
call vcf_tools.intersect_calls as intersect_calls {
input:
lofreq_calls = merge_lofreq.merged_vcf,
muse_calls = muse.muse_vcf,
mutect2_calls = mutect2.filtered_vcf,
strelka_calls = strelka.somatic_snv_vcf,
}
output {
File mutect2_vcf = mutect2.filtered_vcf
File mutect2_filtering_stats = mutect2.filtering_stats
File strelka_snv_vcf = strelka.somatic_snv_vcf
File strelka_indel_vcf = strelka.somatic_indel_vcf
File muse_vcf = muse.muse_vcf
File lofreq_vcf = merge_lofreq.merged_vcf
File intersected_calls = intersect_calls.intersected_calls
}
}
In order to to get them to run concurrently the user reported:
- I moved
Array[String] chr_num = ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "X", "Y", "M"]
Array[String] chrs = prefix("chr", chr_num)
Into defaults.json as "chrs": ["chr1", "chr2", etc.] - I moved LoFreq into a separate workflow and passed it the array chrs
- I complied with dxWDL 1.50. I know this one is deprecated, but I think some of the issues I've been having recently with DNAnexus are coming from dxCompiler.
There appears to be something about the way this sequence of commands is compiled by dxCompiler that prevents it from running in parallel (i.e. muse must run after the scatter) - let me know what you think and what the solution(d) may be.
I think this is the issue discussed in dnanexus/dxWDL#162 where wdl expression evaluations in a WDL workflow gets merged into a fragment for the first following applet.
Referring to the following basic test:
version 1.0
task t1 {
input {
String s
}
command <<<
echo t1 >s
>>>
output {
File out = "s"
}
}
task t2 {
input {
Array[String] s
}
command <<<
echo t1 >s
>>>
output {
File out = "s"
}
}
task dummy {
input {
}
command <<<
>>>
output {
}
}
workflow tinytest3 {
Array[String] chr_num = ["1"]
Array[String] chrs = prefix("chr", chr_num)
call dummy
call t2 {input: s = chrs}
scatter (chr in chrs) {
call t1 { input: s = chr }
}
output {
Array[File] out1 = t1.out
File out2 = t2.out
}
}
With dxCompiler 2.10.0
- The following makes a dnanexus workflow where t2 depends on scatter
workflow tinytest {
Array[String] chr_num = ["1"]
Array[String] chrs = prefix("chr", chr_num)
scatter (chr in chrs) {
call t1 { input: s = chr }
}
call t2 {input: s = chrs}
- The following makes a dnanexus workflow where scatter depends on t2
workflow tinytest2 {
Array[String] chr_num = ["1"]
Array[String] chrs = prefix("chr", chr_num)
call t2 {input: s = chrs}
scatter (chr in chrs) {
call t1 { input: s = chr }
}
- The following makes a dnanexus workflow where scatter and t2 both depend on dummy, but not on each other.
workflow tinytest3 {
Array[String] chr_num = ["1"]
Array[String] chrs = prefix("chr", chr_num)
call dummy
call t2 {input: s = chrs}
scatter (chr in chrs) {
call t1 { input: s = chr }
}
The dependency can be seen by examining the inputs to each stage. E.g. for tinytest3:
$ dx describe workflow-GGGQzJ00Bvbj46PP4gjFyXP9 --json | jq '.stages|.[]|{id: .id,name: .name, input: .input}'
{
"id": "stage-common",
"name": "common",
"input": {}
}
{
"id": "stage-1",
"name": "frag dummy",
"input": {}
}
{
"id": "stage-2",
"name": "t2",
"input": {
"s": {
"$dnanexus_link": {
"outputField": "chrs",
"stage": "stage-1"
}
}
}
}
{
"id": "stage-4",
"name": "scatter (chr in chrs)",
"input": {
"chrs": {
"$dnanexus_link": {
"outputField": "chrs",
"stage": "stage-1"
}
}
}
}
{
"id": "stage-outputs",
"name": "outputs",
"input": {
"t1___out": {
"$dnanexus_link": {
"outputField": "t1___out",
"stage": "stage-4"
}
},
"t2___out": {
"$dnanexus_link": {
"outputField": "out",
"stage": "stage-2"
}
}
}
}
So it's a known behavior that reduces the number of jobs at the expense of potential decrease in concurrency of execution, with a known workaround (see tinytest3).
We can consider making identification of such situations easier or changing the behavior of dxCompiler to place expression evaluation into a separate job when multiple tasks depend on the results of expression evaluation.
@Ben-Habermeyer is there a reason to declare chr_num
and chrs
in the body of the WF, instead of declaring them at the workflow level inputs (with defaults)? This way muse
should depend on a common
stage and not on the scatter.
Hi @Gvaihir thanks for the reply. Yes that is the approach we went with. I think the user was confused why placing the variables in the body of the WF resulted in this behavior
I'll update the doc/Internals.md
with regards to that + recommendations. Short answer - pls avoid declaring variables which don't depend on the outputs of any task/workflow in the body. Allocate those things to the top level inputs - this will eliminate unwanted dependencies. I don't envision any immediate changes to the current logic of dxCompiler with regards to that so will probably close the issue shortly. Otherwise, will update here