Configurable Post-Processing
TimothyWillard opened this issue · 1 comments
Could be useful to have configurable post-processing. A potential way this could work is given below, it allows for both arbitrary command execution and "special" commands for common tasks, but open to suggestions. Basically would create a bash script that would then be executed at the end of a flepiMoP
run. name
would be a human readable name for logging purposes and cmd
is the command. If cmd
is a list construct the arbitrary command and execute that, but if cmd
is a string lookup whatever the "special" command is.
postprocessing:
- name: Foobar
cmd:
- "/usr/bin/python"
- "$FLEPI_PATH/postprocessing/reformat_foobar.py"
- "--arg1"
- "'abc'"
- "--arg2"
- 12345
- name: Plot Predictions
cmd: plot_predictions
- name: AWS S3 Upload
cmd: aws_upload
Current post-processing scripts:
- https://github.com/HopkinsIDD/flepiMoP/blob/main/batch/postprocessing-scripts.sh
- https://github.com/HopkinsIDD/flepiMoP/blob/main/batch/SLURM_postprocess_runner.run
Related to GH-79?
Thanks Tim, I like this idea. We've talked about it before but as always have had no time to think about implementing it.
I think the scripts themselves would have to be rethought a bit to write them in a more optimised way. Currently the main thing I've found is that jobs will time out or run out of memory for a lot of automatic runs if we try to do anything too complicated.