easy_sbatch - Batch submitting Slurm jobs via script templates
git clone https://github.com/shenwei356/easy_sbatch.git
mkdir -p ~/bin
cp easy_sbatch ~/bin/
-
A simple command (no files or data given):
$ easy_sbatch 'cat /etc/hostname' Submitted batch job 15565347 $ ls -rw-r--r-- 1 shenwei cqmu 0 Dec 27 21:50 easy_sbatch.15565347.err -rw-r--r-- 1 shenwei cqmu 9 Dec 27 21:50 easy_sbatch.15565347.out -rw------- 1 shenwei cqmu 314 Dec 27 21:50 easy_sbatch.15565347.slurm -
Handling multiple files and specifying a job name:
$ easy_sbatch 'ls {}' *.fq.gz -J list Submitted batch job 15565371 Submitted batch job 15565373 $ ls -rw-r--r-- 1 shenwei cqmu 0 Dec 27 21:54 list.15565371-read_1.fq.gz.err -rw-r--r-- 1 shenwei cqmu 13 Dec 27 21:54 list.15565371-read_1.fq.gz.out -rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565371-read_1.fq.gz.slurm -rw-r--r-- 1 shenwei cqmu 0 Dec 27 21:55 list.15565373-read_2.fq.gz.err -rw-r--r-- 1 shenwei cqmu 13 Dec 27 21:55 list.15565373-read_2.fq.gz.out -rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565373-read_2.fq.gz.slurm -rw-r--r-- 1 shenwei cqmu 0 Dec 27 20:04 read_1.fq.gz -rw-r--r-- 1 shenwei cqmu 0 Dec 27 20:04 read_2.fq.gz -
From stdin:
$ ls *.fq.gz | easy_sbatch 'echo {/} {%} {%^.fq.gz}' -
Processing paired-ends FASTQ files:
$ ls read_1.fq.gz \ | easy_sbatch 'echo seqtk mergepe {} {^_1.fq.gz}_2.fq.gz' -J pe Submitted batch job 15565379 $ ls -rw-r--r-- 1 shenwei cqmu 0 Dec 27 21:56 pe.15565379-read_1.fq.gz.err -rw-r--r-- 1 shenwei cqmu 40 Dec 27 21:56 pe.15565379-read_1.fq.gz.out -rw------- 1 shenwei cqmu 340 Dec 27 21:56 pe.15565379-read_1.fq.gz.slurm $ cat pe.15565379-read_1.fq.gz.out seqtk mergepe read_1.fq.gz read_2.fq.gz
You can choose another template via the option -t/--template.
You can also edit the value of an option, e.g., replace $partition
with the default partition of your cluster. But note that the
corresponding option from the command line will not take effect
since no variable name is ready to replace.
$ cat ~/.easy_sbatch/default.slurm
#!/bin/bash
#
#SBATCH --partition $partition
#SBATCH --job-name $name
#SBATCH --nodes=1
#SBATCH --cpus-per-task=$ncpus
#SBATCH --mem=$mem
#SBATCH --time=$walltime
#SBATCH --output=$output
#SBATCH --error=$error
cd $$SLURM_SUBMIT_DIR
# echo run on node: $$HOSTNAME >&2
$cmd
usage: easy_sbatch [-h] [-lp | -ls] [-J NAME] [-c NCPUS] [-m MEM] [-p PARTITION] [-w WALLTIME] [-t TEMPLATE]
[-O SCRIPT] [-o OUTPUT] [-e ERROR] [-v]
command [files ...]
easy_sbatch - Batch submitting Slurm jobs with script template
https://github.com/shenwei356/easy_sbatch
Slurm script template:
- Default template is ~/.easy_qsub/default.slurm .
- You can also choose another one via the option -t/--template.
- You can edit the value of an option, e.g., replace "$partition"
with the default partition of your cluster. But note that the
corresponding option from the command line will not take effect
since no variable name is ready to replace.
Supported replacement strings in command:
- "{}" for the full path of a input file.
- "{/}" for dirname.
- "{%}" for basename.
- "{^suffix}" for clipping $suffix.
- "{%^suffix}" for clipping $suffix from basename.
The generated Slurm scripts are saved in /tmp/easy_qsub-user/ by default,
unless the flag -O/--script is set. If jobs are submitted successfully,
the scripts will be moved to the current directory. If not, they will
be removed.
Examples:
1. A simple command (no file or data given):
$ easy_sbatch 'cat /etc/hostname'
$ ls
-rw-r--r-- 1 shenwei cqmu 0 Dec 27 21:50 easy_sbatch.15565347.err
-rw-r--r-- 1 shenwei cqmu 9 Dec 27 21:50 easy_sbatch.15565347.out
-rw------- 1 shenwei cqmu 314 Dec 27 21:50 easy_sbatch.15565347.slurm
2. Handling multiple files and specifying a job name:
$ easy_sbatch 'ls {}' *.fq.gz -J list
$ ls
-rw-r--r-- 1 shenwei cqmu 0 Dec 27 21:54 list.15565371-read_1.fq.gz.err
-rw-r--r-- 1 shenwei cqmu 13 Dec 27 21:54 list.15565371-read_1.fq.gz.out
-rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565371-read_1.fq.gz.slurm
-rw-r--r-- 1 shenwei cqmu 0 Dec 27 21:55 list.15565373-read_2.fq.gz.err
-rw-r--r-- 1 shenwei cqmu 13 Dec 27 21:55 list.15565373-read_2.fq.gz.out
-rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565373-read_2.fq.gz.slurm
-rw-r--r-- 1 shenwei cqmu 0 Dec 27 20:04 read_1.fq.gz
-rw-r--r-- 1 shenwei cqmu 0 Dec 27 20:04 read_2.fq.gz
3. From stdin:
$ ls *.fq.gz | easy_sbatch 'echo {/} {%} {%^.fq.gz}'
4. Processing paired-ends FASTQ files:
$ ls read_1.fq.gz | easy_sbatch 'echo seqtk mergepe {} {^_1.fq.gz}_2.fq.gz' -J pe
$ ls
-rw-r--r-- 1 shenwei cqmu 0 Dec 27 21:56 pe.15565379-read_1.fq.gz.err
-rw-r--r-- 1 shenwei cqmu 40 Dec 27 21:56 pe.15565379-read_1.fq.gz.out
-rw------- 1 shenwei cqmu 340 Dec 27 21:56 pe.15565379-read_1.fq.gz.slurm
$ cat pe.15565379-read_1.fq.gz.out
seqtk mergepe read_1.fq.gz read_2.fq.gz
positional arguments:
command command to submit
files input files
optional arguments:
-h, --help show this help message and exit
-lp, --local_p run commands locally, parallelly
-ls, --local_s run commands locally, serially
-J NAME, --name NAME job name (default: easy_sbatch)
-c NCPUS, --ncpus NCPUS
number of cpus required per task (default: 24)
-m MEM, --mem MEM memory (default: 24gb)
-p PARTITION, --partition PARTITION
partition requested (default: kshdnormal)
-w WALLTIME, --walltime WALLTIME
walltime (default: 30-00:00:00)
-t TEMPLATE, --template TEMPLATE
script template (default: /home/shenwei/.easy_sbatch/default.slurm
-O SCRIPT, --script SCRIPT
output script file
-o OUTPUT, --output OUTPUT
file for batch script's standard output (default: <name>.<jobID>.out])
-e ERROR, --error ERROR
file for batch script's standard error (default: <name>.<jobID>.err])
-v, --verbose verbosely print information. -vv for just printing command not creating scripts and submitting jobs
https://github.com/shenwei356/easy_sbatch
This utility is modified from easy_qsub, which was written for PBS job scheduler.