earthlab/rslurm

No specification on cpus-per-task in the submit.sh

zhiiiyang opened this issue · 2 comments

zhiiiyang commented

Hi, thank you for creating this nice package. However, I do have a question about the output submit.sh file regarding one sbatch argument.

You asked us to specify or by default the number of CPUs per node in the slurm_apply(param, fun, nodes=2, cpus_per_node=2). However, when I checked the submit.sh and found out there is no specification on #Sbatch --cpus-per-task=2. Does it mean that we do not really have controls over how many CPUs per node? Thank you for your help in advance.

Update, I kind of figured it out why it is. It depends on the default value of cpus-per-task under that node. The common value is set to be one, which will put users' under disadvantage since technically they requested more than one core per task.

P.S., I do see them by typing squeue. Please ignore the previous request on that issue.

itcarroll commented

Thanks for the feedback! I cannot claim complete understanding of the sbatch arguments affecting node/task/processor allocation, but I think you've caught a bug. Does the following sound right to you?

  1. Presently, the slurm_apply command passes the cpus_per_node value to the parallel::mcmapply call, effectively instructing each task to use cpus_per_node processors.
  2. The sbatch --cpus-per-task option is not automatically set by slurm_apply, but should be automatically set to the cpus_per_node value.
  3. The slurm_apply option should be renamed to cpus_per_task for consistency with sbatch.
sebschub commented

I am also affected by this issue. Via ConstrainCores=yes in cgroup.conf, our cluster allows only the usage of as many cores on a node as was applied for. The standard value is one core. This means that setting cpus_per_node = 2 in slurm_apply will create two threads with parallel::mcmapply, but each running only with 50% core usage.

What you propose sounds good. Honestly, I don't really understand the difference of --ntasks-per-node and --cpus-per-task, but according to this, the latter and with this your version, is better.