A small library for locking gpus on a single machine.
To run a program on the gpu use:
gpu_run my_program my_args
To see who has locked which gpu run
gpu_lock_info
If somehow your process dies without releasing its lock (gpu_lock_info will tell you if this has happened) then
gpu_lock_release_dead
will release all of your locks where the owning process is no longer running.
If all else fails, go muck about in /var/lock/gpu/, but be careful :)
Install with pip
pip install git+https://github.com/mdenil/gpu_lock
so you can uninstall later with pip uninstall gpu_lock or clone the repo and run python setup.py install.
Create /var/lock/gpu and make sure it is writeable by everyone who will be locking gpus (you need to be root to do this).
sudo mkdir /var/lock/gpu
sudo chmod a+w /var/lock/gpu
Create a file /etc/profile.d/gpu_lock.sh with the following line:
CUDA_VISIBLE_DEVICES=""
export CUDA_VISIBLE_DEVICES
(or add it to /etc/bashrc). This will stop people from accidentally running on an unlocked gpu. Nothing will stop a malicious person from working around this.
You may also want to create /etc/profile.d/gpu_lock.csh with
setenv CUDA_VISIBLE_DEVICES=""