Index Error
Nuyoah0123 opened this issue · 0 comments
Nuyoah0123 commented
Hello, when I was training PMF, an index error occurred halfway through the training. After I changed num_worker=0 and disabled distributed training, the error still existed. What is the cause? The specific error information is as follows:
Traceback (most recent call last):
File "main.py", line 178, in <module>
exp.run()
File "main.py", line 111, in run
self.trainer.run(epoch, mode="Train")
File "/home/LLJ/PMF/tasks/pmf/trainer.py", line 333, in run
for i, (input_feature, input_mask, input_label) in enumerate(dataloader):
File "/home/LLJ/anaconda3/envs/pmf3.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
data = self._next_data()
File "/home/LLJ/anaconda3/envs/pmf3.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
return self._process_data(data)
File "/home/LLJ/anaconda3/envs/pmf3.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
data.reraise()
File "/home/LLJ/anaconda3/envs/pmf3.7/lib/python3.7/site-packages/torch/_utils.py", line 543, in reraise
raise exception
IndexError: Caught IndexError in DataLoader worker process 3.
Original Traceback (most recent call last):
File "/home/LLJ/anaconda3/envs/pmf3.7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "/home/LLJ/anaconda3/envs/pmf3.7/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/LLJ/anaconda3/envs/pmf3.7/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "../../pc_processor/dataset/perspective_view_loader.py", line 184, in __getitem__
proj_tensor = self.aug_ops(proj_tensor)
File "/home/LLJ/anaconda3/envs/pmf3.7/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 95, in __call__
img = t(img)
File "/home/LLJ/anaconda3/envs/pmf3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/home/LLJ/anaconda3/envs/pmf3.7/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 1353, in forward
return F.rotate(img, angle, self.interpolation, self.expand, self.center, fill)
File "/home/LLJ/anaconda3/envs/pmf3.7/lib/python3.7/site-packages/torchvision/transforms/functional.py", line 1118, in rotate
return F_t.rotate(img, matrix=matrix, interpolation=interpolation.value, expand=expand, fill=fill)
File "/home/LLJ/anaconda3/envs/pmf3.7/lib/python3.7/site-packages/torchvision/transforms/functional_tensor.py", line 671, in rotate
return _apply_grid_transform(img, grid, interpolation, fill=fill)
File "/home/LLJ/anaconda3/envs/pmf3.7/lib/python3.7/site-packages/torchvision/transforms/functional_tensor.py", line 573, in _apply_grid_transform
img[mask] = fill_img[mask]
IndexError: index 249108103393 is out of bounds for dimension 2 with size 370
My conda environment is as follows:
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
anyio 3.7.1 pypi_0 pypi
argon2-cffi 23.1.0 pypi_0 pypi
argon2-cffi-bindings 21.2.0 pypi_0 pypi
attrs 24.2.0 pypi_0 pypi
backcall 0.2.0 pypi_0 pypi
beautifulsoup4 4.12.3 pypi_0 pypi
bleach 6.0.0 pypi_0 pypi
ca-certificates 2024.7.2 h06a4308_0
cachetools 5.5.0 pypi_0 pypi
certifi 2022.12.7 py37h06a4308_0
cffi 1.15.1 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
comm 0.1.4 pypi_0 pypi
cycler 0.11.0 pypi_0 pypi
debugpy 1.7.0 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
defusedxml 0.7.1 pypi_0 pypi
descartes 1.1.0 pypi_0 pypi
entrypoints 0.4 pypi_0 pypi
exceptiongroup 1.2.2 pypi_0 pypi
fastjsonschema 2.20.0 pypi_0 pypi
fire 0.6.0 pypi_0 pypi
fonttools 4.38.0 pypi_0 pypi
idna 3.7 pypi_0 pypi
importlib-metadata 6.7.0 pypi_0 pypi
importlib-resources 5.12.0 pypi_0 pypi
ipykernel 6.16.2 pypi_0 pypi
ipython 7.34.0 pypi_0 pypi
ipython-genutils 0.2.0 pypi_0 pypi
ipywidgets 8.1.3 pypi_0 pypi
jedi 0.19.1 pypi_0 pypi
jinja2 3.1.4 pypi_0 pypi
joblib 1.3.2 pypi_0 pypi
jsonschema 4.17.3 pypi_0 pypi
jupyter 1.0.0 pypi_0 pypi
jupyter-client 7.4.9 pypi_0 pypi
jupyter-console 6.6.3 pypi_0 pypi
jupyter-core 4.12.0 pypi_0 pypi
jupyter-server 1.24.0 pypi_0 pypi
jupyterlab-pygments 0.2.2 pypi_0 pypi
jupyterlab-widgets 3.0.11 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1
libffi 3.4.4 h6a678d5_1
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
markupsafe 2.1.5 pypi_0 pypi
matplotlib 3.5.2 pypi_0 pypi
matplotlib-inline 0.1.6 pypi_0 pypi
mistune 3.0.2 pypi_0 pypi
nbclassic 1.1.0 pypi_0 pypi
nbclient 0.7.4 pypi_0 pypi
nbconvert 7.6.0 pypi_0 pypi
nbformat 5.8.0 pypi_0 pypi
ncurses 6.4 h6a678d5_0
nest-asyncio 1.6.0 pypi_0 pypi
notebook 6.5.7 pypi_0 pypi
notebook-shim 0.2.4 pypi_0 pypi
numpy 1.21.6 pypi_0 pypi
nuscenes-devkit 1.1.10 pypi_0 pypi
opencv-python 4.10.0.84 pypi_0 pypi
openssl 1.1.1w h7f8727e_0
packaging 24.0 pypi_0 pypi
pandocfilters 1.5.1 pypi_0 pypi
parso 0.8.4 pypi_0 pypi
pexpect 4.9.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 9.5.0 pypi_0 pypi
pip 22.3.1 py37h06a4308_0
pkgutil-resolve-name 1.3.10 pypi_0 pypi
prometheus-client 0.17.1 pypi_0 pypi
prompt-toolkit 3.0.47 pypi_0 pypi
protobuf 4.24.4 pypi_0 pypi
psutil 6.0.0 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pycocotools 2.0.7 pypi_0 pypi
pycparser 2.21 pypi_0 pypi
pygments 2.17.2 pypi_0 pypi
pyparsing 3.1.2 pypi_0 pypi
pyquaternion 0.9.9 pypi_0 pypi
pyrsistent 0.19.3 pypi_0 pypi
python 3.7.16 h7a1cb2a_0
python-dateutil 2.9.0.post0 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
pyzmq 26.1.0 pypi_0 pypi
qtconsole 5.4.4 pypi_0 pypi
qtpy 2.4.1 pypi_0 pypi
readline 8.2 h5eee18b_0
requests 2.31.0 pypi_0 pypi
scikit-learn 1.0.2 pypi_0 pypi
scipy 1.7.3 pypi_0 pypi
send2trash 1.8.3 pypi_0 pypi
setuptools 65.6.3 py37h06a4308_0
shapely 1.8.5 pypi_0 pypi
six 1.16.0 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
soupsieve 2.4.1 pypi_0 pypi
sqlite 3.45.3 h5eee18b_0
tensorboardx 2.6.2.2 pypi_0 pypi
termcolor 2.3.0 pypi_0 pypi
terminado 0.17.1 pypi_0 pypi
threadpoolctl 3.1.0 pypi_0 pypi
tinycss2 1.2.1 pypi_0 pypi
tk 8.6.14 h39e8969_0
torch 1.13.0+cu117 pypi_0 pypi
torchaudio 0.13.0+cu117 pypi_0 pypi
torchvision 0.14.0+cu117 pypi_0 pypi
tornado 6.2 pypi_0 pypi
tqdm 4.66.5 pypi_0 pypi
traitlets 5.9.0 pypi_0 pypi
typing-extensions 4.7.1 pypi_0 pypi
urllib3 2.0.7 pypi_0 pypi
wcwidth 0.2.13 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
websocket-client 1.6.1 pypi_0 pypi
wheel 0.38.4 py37h06a4308_0
widgetsnbextension 4.0.11 pypi_0 pypi
xz 5.4.6 h5eee18b_1
zipp 3.15.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_1