gbouras13/hybracter

snakemake incompatible to the provided slurm profile

Opened this issue · 4 comments

zhanxw commented

Describe the bug
In snakemake (v8.4.7), "--cluster-status" is not supported, and thus the "cluster-status" in config.yaml will not work.

For more details on this (or incompatible between snakemake and profile), see:
hybracter

aforestsomewhere commented

It seems possible to use the new cluster-executor plugins with Hybracter with a few tweaks, that is working in my hands on the test data.

To add to requirements/recipe (I installed these into the virtual conda environment in which I have installed hybracter):

  • snakemake-minimal=8.10.6
  • snakemake-executor-cluster-generic
  • snakemake-executor-plugin-slurm

config.yaml to be renamed to config.v8+.yaml and contents updated:

  • add "executor: cluster-generic";
  • amend "cluster:" to "cluster-generic-submit-cmd:"
  • amend "cluster-status:" to "cluster-generic-status-cmd:"
  • (optional): add "cluster-generic-cancel-cmd: scancel"
executor: cluster-generic
cluster-generic-submit-cmd:
 mkdir -p logs/{rule} &&
 sbatch
   --nodes=1
   --cpus-per-task={threads}
   --time={resources.time}
   --job-name=smk-{rule}
   --parsable
   --output=logs/{rule}/{jobid}.out
   --error=logs/{rule}/{jobid}.err
#cluster-generic-status-cmd: ~/.config/snakemake/slurm/slurm_status.py
cluster-generic-cancel-cmd: scancel
default-resources:
 - time=1320
jobs: 200
latency-wait: 30
local-cores: 8
restart-times: 1
max-jobs-per-second: 2
max-status-checks-per-second: 10
local-cores: 8
software-deployment-method: conda
rerun-incomplete: True
printshellcmds: True
scheduler: greedy

Sharing in case it is helpful - probably worth testing on other HPC setups (for example I have issues with the sacct command in snakemake workflows locally so could not test the slurm_status.py checker)

  • 👍1
gbouras13 commented

Hi @aforestsomewhere ,

Do you mind if I add this to the documentation eventually - thanks so much for this!

George

  • 👍1
aforestsomewhere commented

Hi @aforestsomewhere ,

Do you mind if I add this to the documentation eventually - thanks so much for this!

George

Of course, glad it was somewhat useful!

Katie

momsane commented

Hi, I was wondering if the latest versions of Hybracter now account for how snakemake v8 uses plugins instead of profiles? I see that in version 0.9.0 snakemake-interface-common, snakemake-interface-executor-plugins, snakemake-interface-report-plugins, snakemake-interface-storage-plugins and snakemake-minimal are included, but not snakemake-executor-cluster-generic
and snakemake-executor-plugin-slurm.