This repository serves as a template for running batch processing jobs on the Roar High-Performance Computing (HPC) system. It includes scripts for job submission, handling dependencies, and organizing output which are tailored for efficient computational research.
This template simplifies the process of submitting and managing dependent jobs on Roar HPC. It employs SLURM for job scheduling, ensuring that jobs are executed in the correct order and only if previous jobs have completed successfully.
- Job Submission Scripts: Includes scripts for submitting jobs via the SLURM scheduler on Roar HPC.
- Dependency Management: Automates the management of job dependencies.
- Output Management: Organizes job outputs and error logs into specified directories, facilitating easy analysis.
Before using this template, ensure you have:
- An active Roar HPC account.
- Basic understanding of SLURM commands.
- Familiarity with Bash scripting and R programming.
- Clone the Repository:
git clone https://github.com/manansaxena/project-template.git cd project-template - Make Scripts Executable:
chmod +x *.sh
- Prepare your scripts: Use 'foo_R.sh' and 'foo.R' as templates and make the necessary changes using the comments in the file. Create as many pair of similar files to run multiple stages in the project.
- Make changes to master_driver.sh: Each stage in the project is assigned a script file to be executed and each stage is executed after the previous one has been completed successfully.
- Create parameters file: Use 'generate_params.R' file to create params.txt which would contain all the combinations you want to run your experiment for. Each stage of the project can have a dedicated parameter file.
After all the changes have been made, just run the following command from your terminal(already connected to Roar):
sbatch master_driver.shYou can check the status of jobs using the following command
squeue -u {your_roar_id}