训练获取Dataloader中 的数据出错

[Junphy-Jan]

报错信息：
Train: data epoch: [0] [ 0/1000] eta: 1:00:53 lr: 0.000001 loss: 0.8702 time: 3.6535 data: 0.0000 max mem: 61442
Traceback (most recent call last):
File "/code/Video-LLaMA/video_llama/datasets/datasets/dataloader_utils.py", line 147, in next
data = next(self.iter_loader)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/root/.vscode-server/extensions/ms-python.python-2023.19.12891009/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 39, in
cli.main()
File "/home/root/.vscode-server/extensions/ms-python.python-2023.19.12891009/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
run()
File "/home/root/.vscode-server/extensions/ms-python.python-2023.19.12891009/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
runpy.run_path(target, run_name="main")
File "/home/root/.vscode-server/extensions/ms-python.python-2023.19.12891009/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
return _run_module_code(code, init_globals, run_name,
File "/home/root/.vscode-server/extensions/ms-python.python-2023.19.12891009/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/home/root/.vscode-server/extensions/ms-python.python-2023.19.12891009/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
exec(code, run_globals)
File "/code/Video-LLaMA/train.py", line 107, in
main()
File "/code/Video-LLaMA/train.py", line 103, in main
runner.train()
File "/code/Video-LLaMA/video_llama/runners/runner_base.py", line 378, in train
train_stats = self.train_epoch(cur_epoch)
File "/code/Video-LLaMA/video_llama/runners/runner_base.py", line 438, in train_epoch
return self.task.train_epoch(
File "/code/Video-LLaMA/video_llama/tasks/base_task.py", line 114, in train_epoch
return self._train_inner_loop(
File "/code/Video-LLaMA/video_llama/tasks/base_task.py", line 205, in _train_inner_loop
samples = next(data_loader)
File "/code/Video-LLaMA/video_llama/datasets/datasets/dataloader_utils.py", line 43, in next
return next(self.loaders[loader_idx])
File "/code/Video-LLaMA/video_llama/datasets/datasets/dataloader_utils.py", line 154, in next
data = next(self.iter_loader)
StopIteration

训练配置

model:
arch: video_llama
model_type: pretrain_llama_v2
freeze_vit: True
freeze_qformer: True

查看 IterLoader 中 self._dataloader 发现 length 为0