Webhooks/events/notifications in HPC PBS scripts

[julianpistorius]

POCs

• 1. Example simple bash script/PBS script which does a callback via curl to a URL. This URL could be Slack, Zapier, IFTTT, etc.
• 2.a. Example of using file watching (entr) to check for a new PBS script in a directory and then submitting it
• 2.b. Example of using file watching (entr) to check for some pattern in a file and then triggering a web hook
• 3. Example of using Python do to some of this instead of using bash
• 4. Example of using psutil in the background to watch resource usage over time and post to Plotly/Google Sheets, etc.

[julianpistorius]

1. Example simple bash script/PBS script which does a callback via curl to a URL.

a. Bash script (test_basic_hook.sh) that does some work and calls a Zapier web hook before and after

Note: Replace the Zapier web hook URL with your own.

curl -v -H "Accept: application/json" \
        -H "Content-Type: application/json" \
        -X POST \
        -d "{\"job_name\":\"$PBS_JOBNAME\",\"job_id\":\"$PBS_JOBID\",\"date\":\"$(date)\",\"event\":\"job_start\"}" \
        https://zapier.com/hooks/catch/99999/88888/

echo "JOBNAME: $PBS_JOBNAME"
echo "JOBID: $PBS_JOBID"

# Note: Long-running work goes here
sleep 10

curl -v -H "Accept: application/json" \
        -H "Content-Type: application/json" \
        -X POST \
        -d "{\"job_name\":\"$PBS_JOBNAME\",\"job_id\":\"$PBS_JOBID\",\"date\":\"$(date)\",\"event\":\"job_end\"}" \
        https://zapier.com/hooks/catch/99999/88888/

b. Example PBS script which calls the script above

#!/bin/bash
        
### Set the Job Name
#PBS -N julianp_webhook_test
        
### Specify the PI group for this job
### List of PI groups available to each user can be found with the "va" command 
#PBS -W group_list=nirav

### Set the queue for this job as windfall or standard (adjust### and #)
### example: PBS -q windfall
#PBS -q standard
##PBS -q windfall

### Set the number of nodes, cores (28 cores / node) and memory (168gb / node)
#PBS -l select=1:ncpus=1:mem=4gb

### Specify "wallclock time" required for this job, hhh:mm:ss
#PBS -l walltime=00:05:00

### Specify total cpu time required for this job, hhh:mm:ss
### total cputime = walltime * ncpus
#PBS -l cput=0:05:00

cd $HOME/webhook_test
./test_basic_hook.sh

c. Zapier 'Zap'

How to Use Zapier Webhooks - https://zapier.com/blog/how-use-zapier-webhooks/

d. Resulting message in Slack

[julianpistorius]

2.a. Example of using file watching (entr) to check for a new PBS script in a directory and then submitting it

a. Install entr: http://entrproject.org/

b. Add script below to ~/bin/run_pbs_check.sh and run it.

#!/bin/bash
trap "exit" INT
mkdir -p $HOME/pbs_inbox
while true; do 
    echo $HOME/pbs_inbox/ | entr -d -p echo /_
    PBS_FILE=$(ls -1 -t $HOME/pbs_inbox/*.pbs 2> /dev/null | head -n 1)
    if [[ -n "$PBS_FILE" ]]; then 
        echo "Submitting file: $PBS_FILE" | tee -a $HOME/pbs_autosubmit.log
        qsub "$PBS_FILE"
        rm "$PBS_FILE"
    else 
        echo "empty"
    fi; 
done

c. Add new PBS files to ~/pbs_inbox/

[agladstein]

Does that automatically submit all pbs scripts that are in the directory $HOME/pbs_inbox?

[julianpistorius]

@agladstein - No, only each new one added after the script is running. One can modify it do that if necessary.