tingofurro/keep_it_simple

utils_misc can't find freer GPU

Opened this issue · 0 comments

The nvidia-smi parsing code results in an empty sequence.

>>> import utils_misc
>>> utils_misc.select_freer_gpu()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/corey/workspace/keep_it_simple/utils_misc.py", line 24, in select_freer_gpu
    freer_gpu = str(get_freer_gpu())
  File "/home/corey/workspace/keep_it_simple/utils_misc.py", line 11, in get_freer_gpu
    return np.argmax(memory_available)
  File "<__array_function__ internals>", line 200, in argmax
  File "/home/corey/workspace/keep_it_simple/.venv/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 1242, in argmax
    return _wrapfunc(a, 'argmax', axis=axis, out=out, **kwds)
  File "/home/corey/workspace/keep_it_simple/.venv/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 54, in _wrapfunc
    return _wrapit(obj, method, *args, **kwds)
  File "/home/corey/workspace/keep_it_simple/.venv/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 43, in _wrapit
    result = getattr(asarray(obj), method)(*args, **kwds)
ValueError: attempt to get argmax of an empty sequence

Changing to 'grep -A5' appears to work for running the sample on a system with one consumer GPU, but I'm not equipped to evaluate the overall impact of the change.

os.system('nvidia-smi -q -d Memory |grep -A5 GPU|grep Free >tmp_smi')