moiexpositoalonsolab/grenepipe

SLURM cluster_config not recognized

Opened this issue · 3 comments

meixilin commented

Hi,

I am trying to run grenepipe with SLURM. When I use the v0.13.1 release with a new install of micromamba, I get this error:

No SLURM account given, trying to guess.
Guessed SLURM account: xxx
No wall time information given. This might or might not work on your cluster. If not, specify the resource runtime in your rule or as a reasonable default via --default-resources.
No job memory information ('mem_mb' or 'mem_mb_per_cpu') is given - submitting without. This might or might not work on your cluster.
....
WorkflowError:
SLURM job submission failed. The error message was sbatch: error: Batch job submission failed: Time limit specification required, but not provided

I think this issue can be fixed with updating the workflow/profiles/slurm/config.yaml file. When I add the following lines to the workflow/profiles/slurm/config.yaml file, my job can start.

default-resources:
    slurm_partition: "xxx"
    slurm_account:   "yyy"
    runtime: 60

However, when I add these lines to cluster_config.yaml, snakemake can't recognize it and have the same error. I think the cluster_config.yaml file might be discontinued?

By the way, other variables (e.g. time needs to become runtime, details here) might have to be updated as well.

Best,
M

lczech commented

Hey Meixi,

thanks for reporting this!

Yes, with grenepipe v0.13.0, we moved to a newer snakemake, which uses a different mechanism to run on slurm, and I have not fully figured out all the changes that this entails... I think most of this is documented somewhere in the slurm executor plugin, but as always with these things just contains the bare minimum (at the time of writing).

So we could definitely add some hand-holding for grenepipe users to make it easier. Your suggestions make sense - and happy to get your feedback on what actually works in practice these days. Feel free to let me know how to change the wiki to accommodate those. Or do you think we also need changes to the grenepipe code itself?

Cheers and so long
Lucas

meixilin commented

Yeah, I think the config file needs some changing. Here are some edits that worked for me.
my edits

lczech commented

Ah nice, yes, that makes sense. I see some interesting changes you made:

  • slurm config fixes
  • --not-recursive option for the generate table script
  • new all_trims shortcut rule
  • trimmomatic settings for ILLUMINACLIP

Those all seem rather useful, thank you! Do you want to create Pull Requests for them? Maybe as individual PRs to keep matters separate? Don't want to just copy those over without giving you the credit for that, so PRs would be the way to become an official contributor to grenepipe :-)

Cheers
Lucas