Error in training ICDAR data on CPU

[mnrao2001]

Hi All,

Sharing my command line for training ICDAR dataset. I'm stuck with the attached error and really struggling to understand on what condition(s) will generator() function terminate in data_processor module.

train.py --gpu_list=0 --input_size=512 --batch_size=10 --nb_workers=1 --training_data_path=data\ICDAR2015\train_data --validation_data_path=data\MLT --checkpoint_path=icdar2015_east_resnet50 --max_epochs=1

After 5 Epochs i get the following error:
...
data\ICDAR2015\train_data\img_2.jpg
data\ICDAR2015\train_data\gt_img_2.txt
Traceback (most recent call last):
File "C:\Python38\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
TypeError: An op outside of the function building code is being passed
a "Graph" tensor. It is possible to have Graph tensors
leak out of the function building context by including a
tf.init_scope in your function building code.
For example, the following function will fail:
@tf.function
def has_init_scope():
my_constant = tf.constant(1.)
with tf.init_scope():
added = my_constant * 2
The graph tensor has name: overly_small_text_region_training_mask:0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "E:\EAST-master\train.py", line 257, in
main()
File "E:\EAST-master\train.py", line 254, in main
history = parallel_model.fit_generator(train_data_generator, epochs=FLAGS.max_epochs, steps_per_epoch=train_samples_count/FLAGS.batch_size, workers=FLAGS.nb_workers, use_multiprocessing=False, max_queue_size=10, callbacks=callbacks, verbose=1)
File "C:\Python38\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "C:\Python38\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1465, in fit_generator
return self.fit(
File "C:\Python38\lib\site-packages\tensorflow\python\keras\engine\training.py", line 66, in _method_wrapper
return method(self, *args, **kwargs)
File "C:\Python38\lib\site-packages\tensorflow\python\keras\engine\training.py", line 848, in fit
tmp_logs = train_function(iterator)
File "C:\Python38\lib\site-packages\tensorflow\python\eager\def_function.py", line 580, in call
result = self._call(*args, **kwds)
File "C:\Python38\lib\site-packages\tensorflow\python\eager\def_function.py", line 644, in _call
return self._stateless_fn(*args, **kwds)
File "C:\Python38\lib\site-packages\tensorflow\python\eager\function.py", line 2420, in call
return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
File "C:\Python38\lib\site-packages\tensorflow\python\eager\function.py", line 1661, in _filtered_call
return self._call_flat(
File "C:\Python38\lib\site-packages\tensorflow\python\eager\function.py", line 1745, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "C:\Python38\lib\site-packages\tensorflow\python\eager\function.py", line 593, in call
outputs = execute.execute(
File "C:\Python38\lib\site-packages\tensorflow\python\eager\execute.py", line 72, in quick_execute
raise core._SymbolicException(
tensorflow.python.eager.core._SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors, but found [<tf.Tensor 'overly_small_text_region_training_mask:0' shape=(None, None, None, 1) dtype=float32>, <tf.Tensor 'text_region_boundary_training_mask:0' shape=(None, None, None, 1) dtype=float32>, <tf.Tensor 'target_score_map:0' shape=(None, None, None, 1) dtype=float32>]