Running a series of batch jobs on a set of machines is a frequent, but annoying, task. Running things manually scales ok up to a few jobs, but beyond that it is hard to track progress. Setting up a proper cluster job management system like Slurm or Sun/Oracle Grid Engine takes a bit of work. Additionally, proper cluster job managers aren't happy with a time-varying set of worker nodes, e.g., if your cluster is built out of AWS EC2 instances or CloudLab nodes. Jawns is designed to bootstrap itself from scratch in under a minute.
The core idea of JAWNS is to use Amazon Web Services to manage the job queue, and all communication to/from worker nodes. All communication takes place via AWS data structures, making communication fast and reliable in the face of network outages. The Jawns programs themselves can then maintain as little state as possible.
- the worker nodes need Java ≥1.8 installed (just the JRE, no JDK needed)
- the
startupcommand needs passwordless-ssh access to each worker node, to launch the worker daemon - the worker nodes, and Jawns commands, need internet access
No shared filesystem (e.g., NFS), or direct communication between Jawns components (e.g., via sockets), is required.
After clong the repo, fill in your AWS credentials (access key and secret key) in src/main/resources/aws-credentials.properties (see the sample file for the required format). These credentials need to grant full access to the AWS services that Jawns uses: SQS, SES, S3 and DynamoDB.
Then build the JAR file via maven:
mvn package
Then create a list of worker node hostnames, one line per file, in workers.txt. Finally, you can run a Jawns command:
java -jar target/jawns-1.0-SNAPSHOT-jar-with-dependencies.jar --workers workers.txt --project test --command status
- Write a Java program that links against the Jawns JAR and calls
boolean submitJobs(List<Job> theJobs)to submit jobs to the cluster. See example code. - Launch worker daemons on the worker nodes via the
startupcommand - Gather results from the worker daemons via the
gathercommand - There are a variety of other commands for checking job status, cancelling jobs, etc. Run Jawns with
--helpfor more details.
There's no need for the job submission, startup, gather or other commands to be run from the same machine, since all state is stored in AWS, making each command essentially stateless. A notable exception is the gather command, which saves output to the local machine.
- send email on job completion/failure
- transfer files to/from workers before/after job completes
- a job can be killed after a timeout is hit
- can create multiple, independent Jawns clusters