###############################################################################
## WeizGrid
## By Stav Yagev, 2013 (Please contact me if you find bugs !)
##
##
## Simple framework for using the SGE cluster on Weizmann for parallel work.
##
## FEATURES:
## - Helps split your own code into fractions that can be run in parallel on the cluster.
## - Allows you to aggregate results in an efficient manner.
## - Write your code ONCE! Execute the same code on PC for debugging and on the cluster
## for production.
## - Recover from errors on the cluster - splitting means if 1 iteration out of a
## 1000 failed you still have 999 iterations in your hand!
##
## Use case example:
## You have a job that processes 1000 images in some way, running the same algorithm
## for each image, outputing something for each image, and finally aggregating the
## results from all images. Up until now, you ran this in a single job that took 1000
## minutes (because lets assume it takes 1 minute to process each image). Using WeizGrid
## you can take advantage of the fact that the job can be parallelized - instead of 1
## job WeizGrid helps you easily split the job to 80 parallel jobs so that you get all
## 1000 images in 1000/80=12.5 minutes (!!)
##
Auto Installation:
==================
1. Copy all the files to ~/WeizGrid
2. In a UNIX terminal, type:
chmod +x ~/WeizGrid/wgsetup
3. Then type:
~/WeizGrid/wgsetup
Usage:
======
1. Create files in the spirit of 'sample.m' and 'calcPrimes.m' (make sure
the WG m-files are in MATLAB's path)
2. Upload your files to UNIX
3. Under UNIX, from the direcotry of YOUR project, run:
~/WeizGrid/qsubWG [name] [your-m-file] [queue]
Example: ~/WeizGrid/qsubWG ParallelCracker sample all.q
4. Enjoy!
Manual Installation (if auto doesn't work...):
=============================================
1. Copy all files to ~/WeizGrid.
2. Make sure all bash scripts have execute permission.
2. Create the following directory structure (1 folder for each queue you intend to use):
~/.matlab/cluster_jobs/all.q
~/.matlab/cluster_jobs/test.q
...
NOTE: The ~/.matlab/ usually already exists but is hidden, so make sure you
are viewing hidden files and folders .
Tips & Troubleshooting:
=======================
- If your algorithm uses a random number, make sure you are not setting the RngShuffle
option to false when invoking WGexec - because this will yield the same random
series for every "parallel" piece of work.
- Sometimes, during the aggregation part of your script, there will be a bug. Then,
you might think all your work is lost! This is not the case, look into part #2
of the documentation of WGgetResults for more information.
- There isn't verbose error checking, so if something doesn't work, check the output
files on your UNIX home for hints... If still unsuccessful, feel free to contact
me for help.