Training Bug: ValueError: need at least one array to stack
dawnKalo opened this issue · 1 comments
dawnKalo commented
It trains the first epoch until the last iteration.
I did it for 10000 iteration as the default.
Then, I tried with 10 epochs to debug it.
For both, it learns until the last iteration:
1/10 [==>...........................] - ETA: 6:06 - loss: 5.8667 - PSNR: 11.3797
2/10 [=====>........................] - ETA: 2:45 - loss: 6.1421 - PSNR: 10.8299
3/10 [========>.....................] - ETA: 1:37 - loss: 6.0711 - PSNR: 10.8295 | 4/10 [00:42<01:25, 14.31s/i 6/10 [=================>............] - ETA: 28s - loss: 6.6172 - PSNR: 10.4240██▌
5/10 [00:42<00:50, 10.17s/i 7/10 [====================>.........] - ETA: 18s - loss: 6.6653 - PSNR: 10.3018██████████▍ |
6/10 [00:43<00:29, 7.27s/i 8/10 [=======================>......] - ETA: 11s - loss: 6.6872 - PSNR: 10.2124██████████████████▎ |
7/10 [00:43<00:15, 5.25s/i 9/10 [==========================>...] - ETA: 4s - loss: 6.7523 - PSNR: 10.1657 ██████████████████████████▏ |
8/10 [00:44<00:07, 3.83s/iEphere (4, 512, 512, 3) PSNR: 10.570: 90%|███████████████████████████████████████████████████████████████████████
| 9/10 [00:44<00:02, 2.84s/iTraceback (most recent call last):
File "main.py", line 240, in <module>
TQDMCallback() | 1/10 [00:40<06:06, 40.73s/i 3/10 [========>.....................] - ETA: 1:37 - loss: 6.0711 - PSNR: 10.8295 | 2/10 [00:41<03:49, 28.67s/i 4/10 [===========>..................] - ETA: 1:03 - loss: 6.3245 - PSNR: 10.5056 | 3/10 [00:41<02:21, 20.22s/i 5/10 [==============>...............] - ETA: 42s - loss: 6.4390 - PSNR: 10.4801 | 4/10 [00:42<01:25, 14.31s/i 6/10 [=================>............] - ETA: 28s - loss: 6.6172 - PSNR: 10.4240██▌ | 5/10 [00:42<00:50, 10.17s/i 7/10 [====================>.........] - ETA: 18s - loss: 6.6653 - PSNR: 10.3018██████████▍ | 6/10 [00:43<00:29, 7.27s/i 8/10 [=======================>......] - ETA: 11s - loss: 6.6872 - PSNR: 10.2124██████████████████▎ | 7/10 [00:43<00:15, 5.25s/i 9/10 [==========================>...] - ETA: 4s - loss: 6.7523 - PSNR: 10.1657 ██████████████████████████▏ | 8/10 [00:44<00:07, 3.83s/iEphere (4, 512, 512, 3) PSNR: 10.570: 90%|███████████████████████████████████████████████████████████████████████ | 9/10 [00:44<00:02, 2.84s/iTraceback (most recent call last):
File "main.py", line 240, in <module>
TQDMCallback()
File "/home/user/code/core/gan_project/PConv-Keras/libs/pconv_model.py", line 254, in fit_generator
*args, **kwargs
File "/home/user/code/core/venv_user/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/user/code/core/venv_user/lib/python3.6/site-packages/keras/engine/training.py", line 1732, in fit_generator
initial_epoch=initial_epoch)
File "/home/user/code/core/venv_user/lib/python3.6/site-packages/keras/engine/training_generator.py", line 242, in fit_generator
workers=0)
File "/home/user/code/core/venv_user/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/user/code/core/venv_user/lib/python3.6/site-packages/keras/engine/training.py", line 1791, in evaluate_generator
verbose=verbose)
File "/home/user/code/core/venv_user/lib/python3.6/site-packages/keras/engine/training_generator.py", line 365, in evaluate_generator
generator_output = next(output_generator)
File "/home/user/code/core/venv_user/lib/python3.6/site-packages/keras/utils/data_utils.py", line 742, in get
six.reraise(*sys.exc_info())
File "/home/user/code/core/venv_user/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/home/user/code/core/venv_user/lib/python3.6/site-packages/keras/utils/data_utils.py", line 711, in get
inputs = future.get(timeout=30)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/home/user/code/core/venv_user/lib/python3.6/site-packages/keras/utils/data_utils.py", line 650, in next_sample
return six.next(_SHARED_SEQUENCES[uid])
File "main.py", line 121, in flow_from_directory
for _ in range(ori.shape[0])], axis=0
File "<__array_function__ internals>", line 6, in stack
File "/home/user/code/core/venv_user/lib/python3.6/site-packages/numpy/core/shape_base.py", line 422, in stack
raise ValueError('need at least one array to stack')
ValueError: need at least one array to stack
Training: 0%| | 0/1 [00:46<?, ?it/s]
Epoch: 0 - loss: 6.752, PSNR: 10.570: 90%|███████████████████████████████████████████████████████████████████████ | 9/10 [00:45<00:05, 5.09s/it]
dawnKalo commented
Found the problem.
The subdirectory of validation should called 'val'.