/qsub_jobs_threads

Sun Grid Engine (SGE) parallel multi-threads jobs submit and monitor program.

Primary LanguagePythonMIT LicenseMIT

qsub_jobs_threads

Makefile as workflow control language

The makefile simply contains rules and a rule consists of three parts: target, (dependencies) and commands. e.g.

soap_align: depend.fq
  soap -p 20 ...other options...

Features

  • Read Makefile and submit parallel multi-threads jobs to Sun Grid Engine (SGE).
  • If user qdel a job in 'qw' state, a period of time(TIMEOUT) later the program will stop.
  • Jobs will be submitted in -pe smp(PE), and log directory will be 'log_SGE'(LOG_DIR).
  • Logs will be append to file log_SGE.out.
  • Specifal comment in the 'target: dependence' line, #-@ INT, will be the threads (INT).
  • No comment (#...) is allowed in command line.

Mechanism:

  • Using 2 objects Parallel_jobs(with nested _Job), and Makefile(with nested _Rule).
  • Using qstat (1st(id) and 5th(state) column) to get jobs ongoing/waiting information.
  • Using qacct (exit_status row) to get jobs stopped(success/fail) information.

Install:

# download.
git clone git@github.com:dazhouze/qsub_jobs_threads.git
# add the directory to environment variable.
export PATH=dir/of/qsub_jobs_threads:$PATH

Run:

qsub_jobs_threads  -f makefile -j #paralell_jobs -q queue.q
# More options will be shown in terminal using:
qsub_jobs_threads -h