Multiple queue logic

[apeltzer]

At the moment, we mostly use a single queue per defined cluster environment. However, it might be that we require more than just a single queue and instead should be able to e.g. define

process A uses short, process B afterward needs much memory mem and process C runs very long, e.g. uses long.

At the moment this is not addressed, but we should be flexible in terms of being able to address this 👍

[jfy133]

I was wondering that. E.g. SHH queues are based on walltimes.

If anyone cleverer than me knows a way for a process to inherit a conditional from the profile, I'd be grateful to learn.

e.g. if ($time > 2.h) { queue = short } else { queue = long }

[ewels]

The nextflow docs has an example very similar to this: https://www.nextflow.io/docs/latest/process.html#dynamic-directives

process foo {

  executor 'sge'
  queue { entries > 100 ? 'long' : 'short' }

  input:
  set entries, file(x) from data

  script:
  """
  < your job here >
  """
}

I guess something similar should work in this case: (untested)

process foo {

  queue { task.time > 2.h ? 'long' : 'short' }

  input:
  set entries, file(x) from data

  script:
  """
  < your job here >
  """
}

[apeltzer]

Can't we add this to generalized nf-core/configs, too? Having something in there that checks whether a given task needs more time than a specified threshold?

e.g. adding this to binac.config that process.queue = { task.time > 2.h ? 'long' : 'short' } or similar for memory / cpus ?

[ewels]

Yeah, that's what I assumed that you wanted to do..

[apeltzer]

I'll try this out 👍 Also, it might be required to change a little the check_max functionality, as this is simply taking the --max_memory for example and having multiple values here might be difficult.

Will give this some thoughts and then maybe submit something here...

[ewels]

I think task variables should be all after the check_max stuff is resolved, hopefully. Not sure that those variables will be available at the config stage though.. 🤔

[apeltzer]

Alright, trying to summarize what I think needs to be considered for stuff like this.

a.) Centralized Configs have a default queue set, e.g. short and would need to be generalized enough to also be able to be used in various pipelines without breaking these.

b.) Having labels would make it possible to specify this kind of stuff, e.g. whenever a process has a label long it will be run on a long queue if the profile selected specifies such a thing. We would need to update all central configs to have the same labels though, e.g. long, short, medium and set these accordingly there to be able to roll this out to all pipelines in the same way.

c.) We'd need to check if check_max() can handle this stuff too. E.g., can we resubmit to long if we failed twice on short due to runtime reasons?

@drpatelh might have some ideas too - @ewels as well but is off the computer atm ;-)

[apeltzer]

We might need to extend the check_max functionality to also have multiple max_cpus and max_memory options for the respective profiles.

An idea:

• have three profiles by default short, medium, long - all available for all centralized configs
• have (max_memory,max_cpus,max_time)x (short,medium,long) as config options in the centralized configs
• extend the check_max function to include the queue argument, making resubmission on failover possible to another queue (add another option to make it possible to configure that a user really wants to do that, e.g. --failover)

max_cpu_short = '16'
max_mem_short = '64.GB'
max_time_short = '48.h'
...
failover = true

This would allow to resubmit jobs on specific error codes to medium and long queues. For users that only have a single queue, simply configure the withLabel:long to be the same for all short/medium/long, so it doesn't fail.

Hope I covered everything, happy for comments by @nf-core/core !

[apeltzer]

@d4straub might be interested as well!

[d4straub]

I am very interested, because https://github.com/nf-core/mag is going to have for one process memory requirements that range from a few GB to a few hundred GB, depending on data and sample composition.

That wide range of requirements cant be handled by one queue on our cluster system but requires access to two different ones (or requires always submitting to the long/high mem queue resulting in huge queuing times).

[apeltzer]

I discussed it with @drpatelh and will probably try this out in a small testcase for a default pipeline (e.g. rnaseq) and report back when this succeeds and/or give feedback what we'd need to change and add to other pipelines or templates without breaking existing functionality. We think this should be possible in general 👍

[d4straub]

Any news here? Also @ggabernet and @skrakau ran into trouble with memory and cluster queues recently!

[ggabernet]

@skrakau we could try this out in binac by adding a binac_smp profile to nf-core/configs and then:
process.queue = { task.memory > XX.GB ? 'smp' : 'short' }

[apeltzer]

One can do that, but that doesn't port between infrastructures - not everyone names their queue smp or short unfortunately 👎

[ewels]

But can you not still have the process.queue config scope within the binac config profile only?

[ggabernet]

yes, exactly that's what we did now, we had process.queue specified inside the binac config profile. We will report if this works fine :)

[apeltzer]

But can you not still have the process.queue config scope within the binac config profile only?

Does that still work with our check_max function that automatically resubmits a job upon failure?

[ewels]

I don't see why not..? It's a separate scope so it shouldn't overwrite anything.. check_max is only used for process.cpus etc, I don't think we really use process.queue anywhere.

[apeltzer]

You are right - I'm not sure what I had in mind before, but that should indeed work!

[maxibor]

I implemented it in the SHH config (#74) and so far it seems to be working :)
process.queue = { task.memory > 756.GB ? 'supercruncher': task.time <= 2.h ? 'short' : task.time <= 48.h ? 'medium': 'long' }

[d4straub]

Works for BinAC as well:
queue = { task.memory >= 128.GB ? 'smp': task.time <= 20.m ? 'tiny' : task.time <= 48.h ? 'short' : task.time <= 168.h ? 'short' : 'long'}

[ewels]

Great! Can we close this issue now?

[ggabernet]

I would say so