GPU memory leak issue
Opened this issue · 6 comments
Hi,
When I use GPU to do face detection, I found that the memory used of the GPU keep increasing. Maybe there is an memory leak issue.
The detector I used as flowing:
mtcnn_detector = MtcnnDetector(model_folder='model', ctx=mx.gpu(0), num_worker = 1, minsize = 40, accurate_landmark = True)
hmm, there are some related issues here:
apache/mxnet#2981
apache/mxnet#3055
looks like it's the problem with the memory manager(naive and pool allocation)...
there's nothing I can do here, but you can change to naivestorgemanager manually , like neodooth did.
我遇到了同样的问题
[16:46:26] /home/zhaixingzi/Applications/mxnet/dmlc-core/include/dmlc/./logging.h:235: [16:46:26] src/storage/./pooled_storage_manager.h:79: cudaMalloc failed: out of memory
--!# guess we have to wait for mxnet's solution.. otherwise you have to revert the memory manager
any suggestion?
after change ptr = new storage::GPUPooledStorageManager(); to ptr = new storage::NaiveStorageManager<storage::GPUDeviceStorage>(); in src/storage/storage.cc line 76
the GPU memory stays in a range ..
but it will consume more memory and takes more time for memory allocation stuff.....
与原版matlab的代码相比，精度会不会下降？
The GPU memory keep increasing