Snakemake won't run more than 2 concurrent jobs even though jobs are "runnable" and there's resource availability
Closed this issue · 2 comments
Describe the Bug
Snakemake workflows with N>2 available nodes limit job execution to only 2 concurrent jobs.
Steps to Reproduce
- Create a snakemake workflow that requires multiple "parallelizable" jobs
- Monitor the logs.
You'll notice that while Snakemake is aware of the existence of the cores, and knows to flag those as "ear marked" for execution (batch status RUNNABLE) it will only ever run 2 concurrent jobs. For convenience, I've created this sample project to reproduce the problem.
Relevant Logs
The sample project above produces this log file. If you scroll down you'll notice (visually) there's only ever 2 logStreamName entries even though all jobs are independent except for the last one.
Expected Behavior
I would expect for more than 2 jobs to run concurrently.
Actual Behavior
Only two jobs are run concurrently.
Screenshots
Additional Context
The project's manifest file and yaml should be both configured to run more than 2 concurrent jobs.
Operating System: macOS 12.3.1
AGC Version: 1.4.0
Was AGC setup with a custom bucket: no
Was AGC setup with a custom VPC: no
I was able to reproduce this problem using spot and non-spot instances.
This issue was sorted out by submitting a request to increase the vCPU limit on my account. The process is fairly straightforward except it took a few days for the changes to reflect.