This is a SPANK plugin for Slurm that can create private temporary
directories for each job. This is useful for /tmp, /var/tmp and other
node-local temporary directories.
The implementation uses unshare(CLONE_NEWNS) to unshare the mount
namespace and then bind mounts the private directories.
If you have issues please let us know or fix it and send us a patch.
Author:
Magnus Jonsson <magnus@hpc2n.umu.se>
Contributions from:
Lars Viklund <lars@hpc2n.umu.se>
Ake Sandgren <ake@hpc2n.umu.se>
Pär Lindfors <paran@nsc.liu.se>
Configuration parameters
------------------------
The SPANK plugin use the following parameters.
base: For each job the plugin will create a directory named
$base.$SLURM_JOB_ID.$SLURM_RESTART_COUNT
mount: Private mount point. This can be specified more than once.
For each mount, a directory will be created in the base dir
and then bind mounted on the specified mount point.
Example
-------
plugstack.conf:
required private-tmpdir.so base=/tmp/slurm mount=/var/tmp mount=/tmp
When a job with jobid 100 and restart count 0, the following
directories will be created on compute nodes:
/tmp/slurm.100.0
/tmp/slurm.100.0/var_tmp
/tmp/slurm.100.0/tmp
In private namespaces the following bind mounts are done:
/tmp/slurm.100.0/var_tmp on /var/tmp
/tmp/slurm.100.0/tmp on /tmp
Note: In this example it is important that mount=/tmp is specified
last, since after /tmp/slurm.100.0/tmp is bind mounted on /tmp
directories in /tmp/slurm.* can no longer be accessed.
Cleanup
-------
The plugin does not do any cleanup.
See below for an example of how to remove temporary directories from
Slurms Epilog script.
Epilog example
--------------
#!/bin/bash
#
# Slurm Epilog script that removes directories created by the
# private-tmpdir SPANK plugin.
#
# TMPDIR_BASE needs to match the configured base= in plugstack.conf
TMPDIR_BASE=/tmp/slurm
PRIVATE_TMPDIR="${TMPDIR_BASE}.${SLURM_JOB_ID}.${SLURM_RESTART_COUNT:=0}"
if [ -d "${PRIVATE_TMPDIR}" ]; then
rm -rf "${PRIVATE_TMPDIR}"
fi