analysiscenter/radio

Pipeline will never stop as n_epochs=None

Closed this issue · 2 comments

Hi guys,

I'm following tutorial 4. Everything works just fine till the very end.
After preprocessing, i define a Keras 3D U-NET, exactly as in the tutorial.
So my code is :


`from radio.models import Keras3DUNet
from radio.models.keras.losses import dice_loss

unet_config = dict(
input_shape = (1, 32, 64, 64),
num_targets = 1,
loss= dice_loss
)

from radio.batchflow import F

train_unet_pipeline = (
combine_crops(cancerset, ncancerset, batch_sizes=(4, 4))
.init_model(
name='3dunet', model_class=Keras3DUNet,
config=unet_config, mode='static'
)
.train_model(
name='3dunet',
x=F(CTIMB.unpack, component='images', data_format='channels_first'),
y=F(CTIMB.unpack, component='masks', data_format='channels_first')
)
)

train_unet_pipeline.run()`


The training begins and seems to work fine bat at the end it stops with the error:


StopIteration Traceback (most recent call last)
~/jupyter/env/lib/python3.7/site-packages/radio/batchflow/batchflow/pipeline.py in _gen_batch(self, *args, **kwargs)
1213 try:
-> 1214 batch_res = self.execute_for(batch)
1215 except SkipBatchException:

~/jupyter/env/lib/python3.7/site-packages/radio/batchflow/batchflow/pipeline.py in execute_for(self, batch, new_loop)
608 batch.pipeline = self
--> 609 batch_res = self._exec_all_actions(batch)
610 batch_res.pipeline = self

~/jupyter/env/lib/python3.7/site-packages/radio/batchflow/batchflow/pipeline.py in _exec_all_actions(self, batch, action_list)
557 elif _action['mode'] == 'n':
--> 558 jbatch = pipe.next_batch()
559 join_batches.append(jbatch)

~/jupyter/env/lib/python3.7/site-packages/radio/batchflow/batchflow/pipeline.py in next_batch(self, *args, **kwargs)
1238 args, kwargs = self._lazy_run
-> 1239 batch_res = self.next_batch(*args, **kwargs)
1240 elif True or kwargs.get('prefetch', 0) > 0:

~/jupyter/env/lib/python3.7/site-packages/radio/batchflow/batchflow/pipeline.py in next_batch(self, *args, **kwargs)
1243 self._batch_generator = self.gen_batch(*args, **kwargs)
-> 1244 batch_res = next(self._batch_generator)
1245 else:

StopIteration:

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last)
in
----> 1 train_unet_pipeline.run()

~/jupyter/env/lib/python3.7/site-packages/radio/batchflow/batchflow/pipeline.py in run(self, init_vars, *args, **kwargs)
1279 warnings.warn('Pipeline will never stop as n_epochs=None')
1280
-> 1281 for _ in self.gen_batch(*args, **kwargs):
1282 pass
1283 return self

RuntimeError: generator raised StopIteration`


So the issue seems easy.
I then try


train_unet_pipeline.run(batch_size=4, n_epochs=2, bar=True)


But i still got the same error, except this time it pops after around 20% of the training.

What am i doing wrong ?

Thanks a lot !

@the-dharma-bum Hey, how do you know how much has it finished training? it doesn't print anything in my console?

Hi, I have the same error.
What is the solution?