Not picking gpu
Closed this issue · 9 comments
Hi,
I am trying to run the example for the RBFE calculations, everything went smooth, but during the production run the somd is not picking up the gpu it is shifting the calculations to the CPU cores any suggestions why it might be happening?
Thank you
Hello there. Could you tell us a little more about your hardware setup? Are you using an NVIDIA GPU, i.e. CUDA compatible? By default, SOMD is set to run on a single GPU and will take the first available device in the CUDA_VISIBLE_DEVICES array from the environment. Could you check what is set, e.g.:
echo $CUDA_VISIBLE_DEVICES
If nothing is set, then the GPU job should fail, but it might be falling back on the CPU for some reason. (Perhaps the scripts select the CPU platform if no GPU is detected. I'm just checking this now.)
hi @mayank-kohli - the tutorial scripts need to be adapted for deployment on your local compute resources. It is possible that your cuda library is compatible with your openmm installation. See e.g. https://github.com/OpenBioSim/biosimspace_tutorials/blob/main/04_fep/02_RBFE/scripts/run_production_slurm.sh
Could you elaborate on what modifications you made to the slurm scripts to make the production run work on your local compute?
@lohedges I am using nvidia GPU A6000 and it is compatible with the CUDA, previously on the same device I have ran multiple openmm jobs and it does run properly.
I looked into it and I found that the jobs are picking up the gpus but for the very little amount of time as you can see in the image below. while the rest of the time it is utilizing cpu.
@jmichel80 I have just commented the module load cuda/gromacs in the production_slurm.sh and rest all is same.
can you post a sample somd.cfg file written by BioSimSpace from one of your production run folders ?
save coordinates = True ncycles = 60 nmoves = 5000 ncycles_per_snap = 4 buffered coordinates frequency = 0 timestep = 2.00 femtosecond reaction field dielectric = 78.3 cutoff type = cutoffperiodic cutoff distance = 10 angstrom barostat = True pressure = 1.00000 atm thermostat = True temperature = 300.00 kelvin inverse friction = 1.00000 picosecond minimise = True constraint = hbonds-notperturbed energy frequency = 250 lambda array = 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 lambda_val = 0.0 perturbed residue number = 23510
Can you try reducing the ncycles and increasing nmoves to compensate, i.e. the total number of integration steps is ncycles * nmoves. Dynamics are only running on the GPU during the cycle. In between coordinates from the OpenMM system are copied back to sire and some post-processing/analysis is done. As such, running few long cycles is the most efficient way to use SOMD.
Hi, reducing the ncycles and increasing the nmoves worked. The GPU utilization has increased.
But I want to understand more on this part can you provide some reference or documentation on this part?
Thank you for your help.
I'm not sure if there is any, perhaps @jmichel80 can point you to something. SOMD was originally designed (I believe) from a Monte Carlo perspective, so you could mix different types of move in cycles, e.g. you could perform a MD cycle, then do some other type of move, etc. If just used for regular MD, as is done here, you really want to run few long cycles in order to achieve optimal performance.
None of the other MD engines supported by BioSImSpace operate in this way, so the default options (generated via a simulation protocol) aren't well optimised for SOMD. The ones set via this tutorial should work well, though, since they've been designed to avoid short cycles.
closed for inactivity