marbl/canu

CANU & Nextflow & SLURM

Closed this issue · 5 comments

hbadrane commented

I'm trying to integrate CANU in a pipeline using Nextflow and run with SLURM. There are some troubles, because CANU is kind of a "wild horse" and Nextflow/Slurm aren't used to such...

  • Using "useGRID=fasle" most likely will solve the problem....
  • But I would like to insist on using Grid, because it automates things and its fast, so here is the problem I'm facing, I'm not an expert in Nextflow nor Slurm, this is how I view it:
    • When the CANU process is started by Nextflow/Slurm, the first task will be running the CANU command (let's name it CANU_cmd)
    • This task "CANU_cmd" configure CANU and starts the grid jobs one by one (CANU_runs)
    • Then CANU_cmd quits and lets the CANU_runs go on...
    • But here is the problem: Nextflow/Slurm think the CANU process is done, and the pipeline tries to proceed to next Process (of course it fails because there is no output yet).

Is there a way to solve this and make CANU_cmd hang-on till the run completes?

skoren commented

It should work if you add gridOptions="--wait" to make the commands Canu submits hold for completion. This does put extra load on the slurm scheduler since there's essentially a busy-wait continuously checking job status and there will be a chain of canu shell scripts waiting on each other (e.g. job 1 waiting on 2 waiting on 3 waiting on 4, etc). This is why the Canu processes run the way they do, there's no busy-waiting like in snakemake/nextflow.

The other option is to use the onSuccess/onFailure options built into Canu, see #2225, #1984.

hbadrane commented

Thanks Sergey.
Will the gridOptions="--wait" also apply to the initial canu command task "canu .... genomesize=xxm ...." ?

skoren commented

Yes, it's a general Canu option for pass-through parameters to the scheduler.

hbadrane commented

Ok I will try it, and also look into the onSuccess/onFailure options.

hbadrane commented

Thank you again, yes the (gridOptions="--wait") solved the problem to have CANU capable of integrating in a pipeline. The onSuccess/onFailure options, haven't tried them yet, but should also work. Very well done and thought of. CLOSING...