test.py运行报错
Closed this issue · 0 comments
Loading checkpoint: /1/dl_pzx/MASTER-pytorch/model_output/master_saved/models/MASTER_Default/example_0314_212753/model_best.pth
with saved best metric 0.9623
test data size: 2 steps: 2
0%| | 0/2 [00:00<?, ?it/s]Fatal Python error: Cannot recover from stack overflow.
Current thread 0x00007fe5b3b2b3c0 (most recent call first):
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/PIL/_util.py", line 6 in isPath
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/PIL/Image.py", line 2971 in open
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 82 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem
...
0%| | 0/2 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/multiprocessing/queues.py", line 104, in get
if not self._poll(timeout):
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/multiprocessing/connection.py", line 414, in _poll
r = wait([self], timeout)
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/multiprocessing/connection.py", line 921, in wait
ready = selector.select(timeout)
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 31156) is killed by signal: Aborted.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/1/dl_pzx/MASTER-pytorch/test.py", line 130, in
predict(args)
File "/1/dl_pzx/MASTER-pytorch/test.py", line 61, in predict
for step_idx, input_data_item in tqdm(enumerate(test_data_loader),total=len(test_data_loader)):
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/tqdm/std.py", line 1185, in iter
for obj in iterable:
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data
idx, data = self._get_data()
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data
success, data = self._try_get_data()
File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1003, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly error is {}'.format(pids_str,e)) from e
RuntimeError: DataLoader worker (pid(s) 31156) exited unexpectedly error is DataLoader worker (pid 31156) is killed by signal: Aborted.
bs设置为了1，num_worker也为1，DataLoader报错的pid也一直在变化，请问是什么原因？