Webhooks/events/notifications in HPC PBS scripts
Opened this issue · 4 comments
julianpistorius commented
POCs
- 1. Example simple bash script/PBS script which does a callback via
curl
to a URL. This URL could be Slack, Zapier, IFTTT, etc. - 2.a. Example of using file watching (
entr
) to check for a new PBS script in a directory and then submitting it - 2.b. Example of using file watching (
entr
) to check for some pattern in a file and then triggering a web hook - 3. Example of using Python do to some of this instead of using bash
- 4. Example of using
psutil
in the background to watch resource usage over time and post to Plotly/Google Sheets, etc.
julianpistorius commented
1. Example simple bash script/PBS script which does a callback via curl to a URL.
a. Bash script (test_basic_hook.sh
) that does some work and calls a Zapier web hook before and after
Note: Replace the Zapier web hook URL with your own.
curl -v -H "Accept: application/json" \
-H "Content-Type: application/json" \
-X POST \
-d "{\"job_name\":\"$PBS_JOBNAME\",\"job_id\":\"$PBS_JOBID\",\"date\":\"$(date)\",\"event\":\"job_start\"}" \
https://zapier.com/hooks/catch/99999/88888/
echo "JOBNAME: $PBS_JOBNAME"
echo "JOBID: $PBS_JOBID"
# Note: Long-running work goes here
sleep 10
curl -v -H "Accept: application/json" \
-H "Content-Type: application/json" \
-X POST \
-d "{\"job_name\":\"$PBS_JOBNAME\",\"job_id\":\"$PBS_JOBID\",\"date\":\"$(date)\",\"event\":\"job_end\"}" \
https://zapier.com/hooks/catch/99999/88888/
b. Example PBS script which calls the script above
#!/bin/bash
### Set the Job Name
#PBS -N julianp_webhook_test
### Specify the PI group for this job
### List of PI groups available to each user can be found with the "va" command
#PBS -W group_list=nirav
### Set the queue for this job as windfall or standard (adjust### and #)
### example: PBS -q windfall
#PBS -q standard
##PBS -q windfall
### Set the number of nodes, cores (28 cores / node) and memory (168gb / node)
#PBS -l select=1:ncpus=1:mem=4gb
### Specify "wallclock time" required for this job, hhh:mm:ss
#PBS -l walltime=00:05:00
### Specify total cpu time required for this job, hhh:mm:ss
### total cputime = walltime * ncpus
#PBS -l cput=0:05:00
cd $HOME/webhook_test
./test_basic_hook.sh
c. Zapier 'Zap'
How to Use Zapier Webhooks - https://zapier.com/blog/how-use-zapier-webhooks/
d. Resulting message in Slack
julianpistorius commented
2.a. Example of using file watching (entr
) to check for a new PBS script in a directory and then submitting it
a. Install entr
: http://entrproject.org/
b. Add script below to ~/bin/run_pbs_check.sh
and run it.
#!/bin/bash
trap "exit" INT
mkdir -p $HOME/pbs_inbox
while true; do
echo $HOME/pbs_inbox/ | entr -d -p echo /_
PBS_FILE=$(ls -1 -t $HOME/pbs_inbox/*.pbs 2> /dev/null | head -n 1)
if [[ -n "$PBS_FILE" ]]; then
echo "Submitting file: $PBS_FILE" | tee -a $HOME/pbs_autosubmit.log
qsub "$PBS_FILE"
rm "$PBS_FILE"
else
echo "empty"
fi;
done
c. Add new PBS files to ~/pbs_inbox/
agladstein commented
Does that automatically submit all pbs scripts that are in the directory $HOME/pbs_inbox?
julianpistorius commented
@agladstein - No, only each new one added after the script is running. One can modify it do that if necessary.