Files and resources to :
- Build Simvascular's svpre, svpost, svsolver on a supercomputing cluster.
- Submit a simulation job using svsolver.
- Check the progress of your simulation
- Download the generated files after the simulation is completed
Note : the following instructions have been written for use on University of Colorado Boulder's research computing account. Please consult your supercomputing facility if you are affiliated with a different cluster, as some steps might be unique for your cluster account system.
You can directly pull this ready built container onto your supercomputing cluster account's project directory (the directory intended to store software builds and smaller data sets) by:
- Login to your cluster account
- go to projects directory
- ssh into compile node
- load singularity module
$ cd projects/$USER/
$ ssh scompile
$ module load singularity/
$ mkdir svsolver-container-build
$ cd svsolver-container-build
$ singularity pull simvascular-svsolver.sif shub://aksa2832/Simvascular-on-cluster
OR
- Create a public github repository and copy the 3 build here files to your github repo. Build files for CU Boulder's supercomputing cluster: quick-build-linux.sh, quick-build-centos7-open-mpi.sh, Singularity (text file containing recipe for the build)
- Then, login to https://singularity-hub.org with your github credentials.
- Go to My collections -> Add collection -> choose your new github repository with the build files and enter the recipe file name as Singularity.
This will automatically build the container for you and place it on singularity hub. Here is how it should be after build:
- Download the container to your supercomputing cluster account's project directory (the directory intended to store software builds and smaller data sets) by:
$ cd projects/$USER/
$ ssh scompile
$ module load singularity/
$ mkdir svsolver-container-build
$ cd svsolver-container-build
$ singularity pull simvascular-svsolver.sif shub://<yourgithubusername>/<svsolver_github_reponame>
Step 1: Create the data files for the job with Simvascular application on your local computer.
- Open SimVascular
- Right click the job node (eg. "test-simvascular-project") under Simulations in Data Manager
- Click "Export Data Files"
- Select a folder for exporting(eg. "test-simvascular-project-data").
Step 2: Write and load a solver script.
- Write the script: Follow instructions given in the template file "sample_jobscript" and modify the parameters according to your requirements.
- Save the script as your-job-script.sh to the folder containing the exported data files from SimVascular after step 1("test-simvascular-project-data") . This folder should then contain all the Data Files and the job script required for the simulation.
Step 3: Upload this data-folder on your supercomputing cluster account's working directory(usually /scratch/$USER).
- Open terminal(or equivalent command prompt).
- Secure copy the data-folder("test-simvascular-project-data") onto the cluster's working directory. Here @login.rc.colorado.edu is the login id you use for cluster account.
$ scp -rv </path/to/test-simvascular-project-data/on/your/computer> <username>@login.rc.colorado.edu:</target/path/to/working/directory/on/cluster/>
Step 4: Submit the simulation job.
- Login to your cluster account
- navigate to the working directory(eg. /scratch/$USER/test-simvascular-project-data) from where the simulation will be launched.
- Enter the following commands to submit the job:
$ module load slurm/summit
$ sbatch your-job-script.sh
- If you have enabled email notification in you job script, you will be notified as soon as the simulation starts and the moment it ends via. email.
- Find information on your job’s start time using the squeue command:
$ squeue --user=your_rc-username --start
- To check the status/progress of the submitted job
$ sacct --starttime=YYYY-MM-DD --format=User,JobName,JobId,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
- Finally check the output file(.out file) generated in </scratch/$USER/test-simvascular-project-data> for simulation performance details.
- Compress the folder containing generated files(will be named n-procs_case for n cores used):
$ tar -cvzf <<n-cores>-procs_case>.tar.gz </scratch/summit/$USER/test-simvascular-project-data/<n-cores>-procs_case>
- Use globus file transfer or secure copy the zipped folder on your local computer:
scp -rv <username>@login.rc.colorado.edu:</scratch/summit/$USER/test-simvascular-project-data/<n-cores>-procs_case.tar.gz> <target/storage/path/on/your/computer>