Running tensorflow backend on single gpu but on multi gpu machine
oak-tree opened this issue · 11 comments
This result with:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.39 Driver Version: 375.39 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K80 Off | 8CFB:00:00.0 Off | 0 |
| N/A 58C P0 58W / 149W | 10873MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K80 Off | 9657:00:00.0 Off | 0 |
| N/A 74C P0 65W / 149W | 10873MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla K80 Off | A78F:00:00.0 Off | 0 |
| N/A 42C P0 71W / 149W | 10871MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla K80 Off | BB06:00:00.0 Off | 0 |
| N/A 67C P0 141W / 149W | 10941MiB / 11439MiB | 58% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 4543 C python 10869MiB |
| 1 4543 C python 10869MiB |
| 2 4543 C python 10867MiB |
| 3 4543 C python 10937MiB |
+-----------------------------------------------------------------------------+
As can be seen above: Only 1 gpu is being used. But the memory has been allocated for all 4 gpu
Any idea how to prevent it?
This is default behavior for TensorFlow
If you want TensorFlow to not allocate memory for "all of the GPUs", add the following to your script:
import os
os.environ["CUDA_VISIBLE_DEVICES"]="0"
This will make it so that only the 0th GPU is visible to TensorFlow.
In your case, you can choose any in the range [0, 3].
If you wanted, for example, the 0th and 2nd GPUs to both be visible, replace "0" with "0,2".
If you want TensorFlow to not allocate "all of the memory" for the GPUs visible to it, then add:
from keras import backend as K
config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)
K.set_session(sess)
Hope this helps!
Hey thanks for the information. I'll check it out. While this is not tensorflow board, can you still explain why does it allocate all GPU even if it uses only a single one?
You're welcome!
While TensorFlow utilizes only one GPU by default, I believe it allocates the full memory of all of the GPUs such that the user can readily [that is, front loading the setup time] use the TensorFlow interface to incorporate use of the other GPUs in their training/inference pipelines.
Sorry if that's a bit of a non-answer, but maybe these links will help:
TensorFlow notes on device placement with GPUs
Keras blog notes on multi-GPU training
Thanks!
I am having the same issue. I am using Keras R version, do you know to do this in R version?
Set the environment variable in your shell to
CUDA_VISIBLE_DEVICES=0
import os
os.environ["CUDA_VISIBLE_DEVICES"]="0,1,2"
config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)
keras.backend.set_session(sess)
def _get_available_devices():
from tensorflow.python.client import device_lib
local_device_protos = device_lib.list_local_devices()
return [x.name for x in local_device_protos]
print _get_available_devices()Results in
u'/device:CPU:0', u'/device:GPU:0'
The CUDA_VISIBLE_DEVICES solution ignores the possibility that one is running stuff on a queue, where one cannot know at runtime which device will be used! Has maybe a better solution appeared in the three years that tensorflow already exists?
Same issue, But I want to know, how to release the rest gpus if tf session has started, without killing process and restarting it , thank you!
Same question, how to set a GPU per python thread. An environment variable is nice for setting the visdisle GPU for the whole script, but what if I have two threads in the same script, consuming a queue, each controlling their own GPU?
set a GPU per python thread
Same question here!!! I have google a lot but have not found any good solution. Any help will be greatly appreciated!