workflow failure w/ cmd exit status 0 and 2 (w/ cmd error exit status 125)

[mjpworth]

tagging @mikek498

nextflow.1.log
nextflow.2.log
I have two separate workflow runs on the same data set that both fail, but in two different spots.

the first run fails with:

Error executing process > 'ncovIllumina:sequenceAnalysis:makeQCCSV (19COV000059-NYC-M04731-200425_S27_L001)'
Caused by:
Missing output file(s) 19COV000059-NYC-M04731-200425_S27_L001.depth.png expected by process ncovIllumina:sequenceAnalysis:makeQCCSV (19COV000059-NYC-M04731-200425_S27_L001)

Command executed:

qc.py --illumina --outfile nyc.19COV000059-NYC-M04731-200425_S27_L001.qc.csv --sample 19COV000059-NYC-M04731-200425_S27_L001 --ref nCoV-2019.reference.fasta --bam 19COV000059-NYC-M04731-200425_S27_L001.mapped.primertrimmed.sorted.bam --fasta 19COV000059-NYC-M04731-200425_S27_L001.primertrimmed.consensus.fa

Command exit status:
0

Command output:
(empty)

the attached 'nextflow.1.log' file is the log file of this run

the second run fails with:

Error executing process > 'ncovIllumina:sequenceAnalysis:makeConsensus (19COV000038-NYC-M04731-200425_S6_L001)'

Caused by:
Process ncovIllumina:sequenceAnalysis:makeConsensus (19COV000038-NYC-M04731-200425_S6_L001) terminated with an error exit status (2)

Command executed:

samtools mpileup -A -d 100000 -Q0 19COV000038-NYC-M04731-200425_S6_L001.mapped.primertrimmed.sorted.bam | ivar consensus -t 0.75 -m 50 -n N -p 19COV000038-NYC-M04731-200425_S6_L001.primertrimmed.consensus

Command exit status:
2

Command output:
(empty)
Command error:
Execution failed: generic::unknown: action 2: waiting for container: starting container: running ["docker" "start" "-ai" "b8045f1787ede787ac23d8964d628bb99c26c1337b5c9ba0eefa215ce2277264"]: exit status 125

the attached 'nextflow.2.log' file is the log file of this run

My apologies for not offering much troubleshooting here... frankly, i'm not sure how to approach this. thanks very much for your efforts and attention.

[mjpworth]

I've found a work-around or a solution... not sure which. This fix was suggested by staff at Seqera.io.

In the gls.config file, I change this line:
FROM: process.errorStrategy = { task.exitStatus==14 ? 'retry' : 'terminate' }
TO: process.errorStrategy = 'retry'

effectively forcing the workflow to retry a process on any error exit code, not just 14. This was enough to get my workflows to run to completion. so... apparently simply retrying the failed process is enough.

just letting @mikek498 know too.

thanks,
@mjpworth