utils_misc can't find freer GPU
Opened this issue · 0 comments
corey-cole commented
The nvidia-smi parsing code results in an empty sequence.
>>> import utils_misc
>>> utils_misc.select_freer_gpu()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/corey/workspace/keep_it_simple/utils_misc.py", line 24, in select_freer_gpu
freer_gpu = str(get_freer_gpu())
File "/home/corey/workspace/keep_it_simple/utils_misc.py", line 11, in get_freer_gpu
return np.argmax(memory_available)
File "<__array_function__ internals>", line 200, in argmax
File "/home/corey/workspace/keep_it_simple/.venv/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 1242, in argmax
return _wrapfunc(a, 'argmax', axis=axis, out=out, **kwds)
File "/home/corey/workspace/keep_it_simple/.venv/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 54, in _wrapfunc
return _wrapit(obj, method, *args, **kwds)
File "/home/corey/workspace/keep_it_simple/.venv/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 43, in _wrapit
result = getattr(asarray(obj), method)(*args, **kwds)
ValueError: attempt to get argmax of an empty sequence
Changing to 'grep -A5' appears to work for running the sample on a system with one consumer GPU, but I'm not equipped to evaluate the overall impact of the change.
os.system('nvidia-smi -q -d Memory |grep -A5 GPU|grep Free >tmp_smi')