RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 1289945088 bytes. Error code 12 (Cannot allocate memory)
apanand14 opened this issue · 5 comments
Hello,
I'm trying to preapre datset for running BPR. But I'm facing this error. Can you please help me out in this matter? thank you in advance.
**Traceback (most recent call last):
File "/home/Anaconda3/envs/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, kwds))
File "./tools/split_patches.py", line 199, in run_inst
dets = get_dets(maskdt, args.patch_size, args.iou_thresh)
File "./tools/split_patches.py", line 102, in get_dets
fbmask = find_float_boundary(maskdt)
File "./tools/split_patches.py", line 55, in find_float_boundary
stride=1, padding=width//2).permute(1, 0, 2, 3)
RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 1289945088 bytes. Error code 12 (Cannot allocate memory)
It seems that the (CPU) memory is insufficient. We tested on a machine with 188G memory.
Thank you for your answer! I somehow managed to solve this by creating another enviornment and installing everything again. This might be the multiprocessing error and might be corrupted so it threw this error. But thank you!
@tinyalpha I got another issue in the last stage of inference( merge patches). I have successfully created .pkl file but when i run merge patches then I face this error. if possible then please look into this. Thank you in advance.
Traceback (most recent call last):
File "./tools/merge_patches.py", line 104, in
start()
File "./tools/merge_patches.py", line 74, in start
for r in p.imap_unordered(run_inst, enumerate(dt)):
File "/home/Anaconda3/envs/closemmlab/lib/python3.8/multiprocessing/pool.py", line 868, in next
raise value
KeyError: 1300000
(closemmlab) []$ python tools/merge_patches.py refinemask_r50.val.segm.json /Data/annotations/instances_val2017.json refinemask_r50/refined.pkl refinemask_r50/patches/detail_dir/val refinemask_r50/refined.json
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
0%| | 0/679 [00:00<?, ?it/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/Anaconda3/envs/closemmlab/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "tools/merge_patches.py", line 36, in run_inst
patch_mask = results[pid]
KeyError: 200000
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "tools/merge_patches.py", line 104, in
start()
File "tools/merge_patches.py", line 74, in start
for r in p.imap_unordered(run_inst, enumerate(dt)):
File "/home/Anaconda3/envs/closemmlab/lib/python3.8/multiprocessing/pool.py", line 868, in next
raise value
KeyError: 200000
(closemmlab) []$ python tools/merge_patches.py refinemask_r50.val.segm.json /Data/annotations/instances_val2017.json refinemask_r50/refined.pkl refinemask_r50/patches/detail_dir/val refinemask_r50/refined.json
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
0%| | 0/679 [00:00<?, ?it/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/Anaconda3/envs/closemmlab/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "tools/merge_patches.py", line 36, in run_inst
patch_mask = results[pid]
KeyError: 700000
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "tools/merge_patches.py", line 104, in
start()
File "tools/merge_patches.py", line 74, in start
for r in p.imap_unordered(run_inst, enumerate(dt)):
File "/home/Anaconda3/envs/closemmlab/lib/python3.8/multiprocessing/pool.py", line 868, in next
raise value
KeyError: 700000
This may be because the .pkl file and refinemask_r50/patches/detail_dir/val do not match.
The results of these patches (indicated by these pids) are missing in the .pkl file.
Thank you for answer. I created patches again and it worked. Thanks once again.