earthlab/rslurm

slurm_options for effectively disabling mcmapply?

jgrn307 opened this issue · 2 comments

jgrn307 commented

I'm using randomForestSRC within an rslurm loop, and it has a reported behavior of not behaving when mcmapply is "active" (it uses OpenMPI which has some conflicts with parallel). What I was wondering is if there is a combination of slurm_options and nodes and cpus_per_node that effectively passes the "work" of parallelizing WITHIN a node to the slurm manager, vs. using mcmapply to do it. Basically: is there a way to allow each Rscript to run restricted to a SINGLE cpu on a SINGLE node but within an array job? Right now, I can of course set cpus-per-node to 1 to basically "disable" mcmapply, but then the other cpus are "idle" on that node, so its a waste of resources. Thoughts?

itcarroll commented

to get N processes, each running as a single task using 1 cpu (and thwarting mcmapply), try ...

slurm_apply( ...
  cpus_per_node = 1
  nodes = N
  slurm_options = list("cpu-per-task" = 1)

this is a guess. whether multiple tasks, even if they are only using 1 cpu, are allowed to run on one node may depend on your Slurm configuration.

qdread commented

As @itcarroll says, whether or not multiple array tasks can run independently on different CPUs on the same node may depend on the user's Slurm configuration. Addressing this is probably outside the purview of rslurm. Closing.