training: Cannot convert a symbolic Tensor to a numpy array
bertsky opened this issue · 7 comments
bertsky commented
I have Calamari 2.2.1 / Python 3.7.4 / TF 2.4.1 / Numpy 1.21.6, and attempt calamari-train in a (Power9-based) HPC environment.
Unfortunately, I get the following exception:
stacktrace
INFO 2022-04-26 18:48:37,860 tfaip.trainer.trainer: Setting samples per epoch relative to dataset size with a factor of 1.
INFO 2022-04-26 18:48:37,860 tfaip.trainer.trainer: Set steps per epoch to 18
INFO 2022-04-26 18:48:37,874 tfaip.scenario.scenariobase: Building training keras model
INFO 2022-04-26 18:48:37,881 tfaip.scenario.scenariobase: Attempting to set no train scope
INFO 2022-04-26 18:48:37,881 tfaip.scenario.scenariobase: Compiling training model including optimization
INFO 2022-04-26 18:48:37,892 tfaip.scenario.scenariobase: Interference on the input shapes to build the graph.
CRITICAL 2022-04-26 18:48:39,339 tfaip.util.logging: Uncaught exception
Traceback (most recent call last):
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/scripts/train.py", line 29, in main
return trainer.train()
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/training/trainer.py", line 190, in train
last_logs = super().train(callbacks=callbacks)
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/device/device_config.py", line 134, in wrapper
return func(*args, **kwargs)
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/trainer/trainer.py", line 209, in train
training_graph_only=self._training_graph_only,
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/scenario/scenariobase.py", line 627, in setup_training
self._keras_train_model((real_inputs, wrapped_targets, real_meta))
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 952, in __call__
input_list)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1091, in _functional_construction_call
inputs, input_masks, args, kwargs)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 822, in _keras_tensor_symbolic_call
return self._infer_output_signature(inputs, args, kwargs, input_masks)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 863, in _infer_output_signature
outputs = call_fn(inputs, *args, **kwargs)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 670, in wrapper
raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in user code:
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:72 call *
outputs = self.graph.train(inputs, pre_proc_targets)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:272 train *
return self(wrap((inputs, targets)))
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:267 call *
return self._graph(inputs_targets)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:312 call *
outputs = self.build_train_graph(inputs, targets, training=training)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:343 build_train_graph *
return self.build_graph(inputs, training)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/graph.py:83 build_graph *
last_layer_output = layer(last_layer_output)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/layer.py:77 call *
return self._call(inputs, **kwargs)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/bilstm.py:50 _call *
return self.lstm(inputs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:539 __call__ **
return super(Bidirectional, self).__call__(inputs, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__
outputs = call_fn(inputs, *args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:653 call
initial_state=forward_state, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:660 __call__
return super(RNN, self).__call__(inputs, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__
outputs = call_fn(inputs, *args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent_v2.py:1157 call
inputs, initial_state, _ = self._process_inputs(inputs, initial_state, None)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:859 _process_inputs
initial_state = self.get_initial_state(inputs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:643 get_initial_state
inputs=None, batch_size=batch_size, dtype=dtype)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2507 get_initial_state
self, inputs, batch_size, dtype))
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2987 _generate_zero_filled_state_for_cell
return _generate_zero_filled_state(batch_size, cell.state_size, dtype)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:3003 _generate_zero_filled_state
return nest.map_structure(create_zeros, state_size)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659 map_structure
structure[0], [func(*x) for x in entries],
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659 <listcomp>
structure[0], [func(*x) for x in entries],
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:3000 create_zeros
return array_ops.zeros(init_state_size, dtype=dtype)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
return target(*args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2819 wrapped
tensor = fun(*args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2868 zeros
output = _constant_if_small(zero, shape, dtype, name)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2804 _constant_if_small
if np.prod(shape) < 1000:
<__array_function__ internals>:6 prod
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3052 prod
keepdims=keepdims, initial=initial, where=where)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/numpy/core/fromnumeric.py:86 _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:855 __array__
" a NumPy call, which is not supported".format(self.name))
NotImplementedError: Cannot convert a symbolic Tensor (root/root/CalamariGraph/lstm_0/bidirectional/forward_lstm/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
CRITICAL 2022-04-26 18:48:39,375 tfaip.util.logging: Uncaught exception
Traceback (most recent call last):
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/bin/calamari-train", line 8, in <module>
sys.exit(run())
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/scripts/train.py", line 17, in run
main(parse_args())
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/scripts/train.py", line 29, in main
return trainer.train()
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/training/trainer.py", line 190, in train
last_logs = super().train(callbacks=callbacks)
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/device/device_config.py", line 134, in wrapper
return func(*args, **kwargs)
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/trainer/trainer.py", line 209, in train
training_graph_only=self._training_graph_only,
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/scenario/scenariobase.py", line 627, in setup_training
self._keras_train_model((real_inputs, wrapped_targets, real_meta))
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 952, in __call__
input_list)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1091, in _functional_construction_call
inputs, input_masks, args, kwargs)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 822, in _keras_tensor_symbolic_call
return self._infer_output_signature(inputs, args, kwargs, input_masks)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 863, in _infer_output_signature
outputs = call_fn(inputs, *args, **kwargs)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 670, in wrapper
raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in user code:
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:72 call *
outputs = self.graph.train(inputs, pre_proc_targets)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:272 train *
return self(wrap((inputs, targets)))
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:267 call *
return self._graph(inputs_targets)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:312 call *
outputs = self.build_train_graph(inputs, targets, training=training)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:343 build_train_graph *
return self.build_graph(inputs, training)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/graph.py:83 build_graph *
last_layer_output = layer(last_layer_output)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/layer.py:77 call *
return self._call(inputs, **kwargs)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/bilstm.py:50 _call *
return self.lstm(inputs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:539 __call__ **
return super(Bidirectional, self).__call__(inputs, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__
outputs = call_fn(inputs, *args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:653 call
initial_state=forward_state, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:660 __call__
return super(RNN, self).__call__(inputs, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__
outputs = call_fn(inputs, *args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent_v2.py:1157 call
inputs, initial_state, _ = self._process_inputs(inputs, initial_state, None)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:859 _process_inputs
initial_state = self.get_initial_state(inputs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:643 get_initial_state
inputs=None, batch_size=batch_size, dtype=dtype)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2507 get_initial_state
self, inputs, batch_size, dtype))
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2987 _generate_zero_filled_state_for_cell
return _generate_zero_filled_state(batch_size, cell.state_size, dtype)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:3003 _generate_zero_filled_state
return nest.map_structure(create_zeros, state_size)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659 map_structure
structure[0], [func(*x) for x in entries],
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659 <listcomp>
structure[0], [func(*x) for x in entries],
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:3000 create_zeros
return array_ops.zeros(init_state_size, dtype=dtype)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
return target(*args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2819 wrapped
tensor = fun(*args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2868 zeros
output = _constant_if_small(zero, shape, dtype, name)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2804 _constant_if_small
if np.prod(shape) < 1000:
<__array_function__ internals>:6 prod
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3052 prod
keepdims=keepdims, initial=initial, where=where)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/numpy/core/fromnumeric.py:86 _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:855 __array__
" a NumPy call, which is not supported".format(self.name))
NotImplementedError: Cannot convert a symbolic Tensor (root/root/CalamariGraph/lstm_0/bidirectional/forward_lstm/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
Probably the same as #314.
bertsky commented
Note: I was trying to warmstart from calamari_models_experimental/deep3_fraktur19/0.ckpt.json.
Is there a known workaround?
andbue commented
Sorry, I've never encountered it myself. Since it disappeared in #314 after a reinstall, could you provide some more information on how you installed calamari?
Here is a dockerfile that I use on our hpc cluster, maybe that helps:
FROM nvidia/cuda:11.5.1-cudnn8-runtime-ubuntu18.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y --no-install-recommends software-properties-common && \
add-apt-repository -y ppa:ubuntu-toolchain-r/ppa && \
apt-get purge -y --auto-remove software-properties-common && \
apt-get install -y --no-install-recommends build-essential && \
apt-get install -y --no-install-recommends python3.7 python3.7-dev python3-pip python3-setuptools python3-wheel python3-distutils && \
apt clean && \
python3.7 -m pip --no-cache-dir install --upgrade pip && \
python3.7 -m pip --no-cache-dir install https://github.com/Calamari-OCR/calamari/tarball/master && \
apt-get purge -y --auto-remove build-essential && \
rm -rf /var/lib/apt/lists/*bertsky commented
I installed from PyPI. Just tried --force-reinstall to no avail. Also tried updating, which gave me 2.2.2 intead of 2.2.1, but still no change (the error persists).
pip log
pip install -U calamari-ocr
Requirement already satisfied: calamari-ocr in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (2.2.1)
Collecting calamari-ocr
Using cached calamari_ocr-2.2.2-py3-none-any.whl (198 kB)
Requirement already satisfied: python-bidi in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from calamari-ocr) (0.4.2)
Requirement already satisfied: tensorflow>=2.4.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from calamari-ocr) (2.4.1)
Requirement already satisfied: lxml in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from calamari-ocr) (4.8.0)
Requirement already satisfied: h5py in /software/ml/h5py/2.10.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/h5py-2.10.0-py3.7-linux-ppc64le.egg (from calamari-ocr) (2.10.0)
Requirement already satisfied: edit-distance in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from calamari-ocr) (1.0.4)
Requirement already satisfied: tfaip==1.2.6 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from calamari-ocr) (1.2.6)
Requirement already satisfied: xlsxwriter in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from calamari-ocr) (3.0.3)
Requirement already satisfied: pooch==1.4.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (1.4.0)
Requirement already satisfied: GitPython in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (3.1.27)
Requirement already satisfied: python-Levenshtein in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (0.12.2)
Requirement already satisfied: prettytable in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (3.2.0)
Requirement already satisfied: pandas in /software/ml/SciPy-bundle/2019.10-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (0.25.3)
Requirement already satisfied: dataclasses-json in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (0.5.5)
Requirement already satisfied: imageio in /software/ml/scikit-image/0.16.2-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (2.6.1)
Requirement already satisfied: opencv-python-headless in /home/h1/rosa992c/.local/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (4.2.0+d608d69)
Requirement already satisfied: openpyxl in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (3.0.9)
Requirement already satisfied: nptyping in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (2.0.0)
Requirement already satisfied: xlrd==1.2.0 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (1.2.0)
Requirement already satisfied: paiargparse==1.1.2 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (1.1.2)
Requirement already satisfied: scikit-image in /software/ml/scikit-image/0.16.2-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (0.16.2)
Requirement already satisfied: typeguard in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (2.13.3)
Requirement already satisfied: pillow in /home/h1/rosa992c/.local/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (7.2.0)
Requirement already satisfied: adabelief-tf in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (0.2.1)
Requirement already satisfied: tqdm in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (4.64.0)
Requirement already satisfied: tensorflow-addons>=0.12.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (0.12.1)
Requirement already satisfied: editdistance in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from paiargparse==1.1.2->tfaip==1.2.6->calamari-ocr) (0.6.0)
Requirement already satisfied: marshmallow-enum<2.0.0,>=1.5.1 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from dataclasses-json->tfaip==1.2.6->calamari-ocr) (1.5.1)
Requirement already satisfied: typing-inspect>=0.4.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from dataclasses-json->tfaip==1.2.6->calamari-ocr) (0.7.1)
Requirement already satisfied: marshmallow<4.0.0,>=3.3.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from dataclasses-json->tfaip==1.2.6->calamari-ocr) (3.15.0)
Requirement already satisfied: appdirs in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (1.4.4)
Requirement already satisfied: packaging in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (19.1)
Requirement already satisfied: requests in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (2.22.0)
Requirement already satisfied: numpy>=1.16.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.21.6)
Requirement already satisfied: keras-preprocessing~=1.1.2 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.1.2)
Requirement already satisfied: opt-einsum>=3.3.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (3.3.0)
Requirement already satisfied: grpcio~=1.32.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.32.0)
Requirement already satisfied: absl-py~=0.10 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (0.10.0)
Requirement already satisfied: wrapt>=1.12.1 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.12.1)
Requirement already satisfied: termcolor>=1.1.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.1.0)
Requirement already satisfied: tensorflow-estimator<2.5.0,>=2.4.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (2.4.0)
Requirement already satisfied: google-pasta>=0.2 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (0.2.0)
Requirement already satisfied: flatbuffers>=1.12.0 in /software/ml/flatbuffers-python/1.12-GCCcore-8.3.0-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.12)
Requirement already satisfied: six>=1.12.0 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.12.0)
Requirement already satisfied: typing-extensions>=3.7.4.2 in /software/ml/typing-extensions/3.7.4.3-GCCcore-8.3.0-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (3.7.4.3)
Requirement already satisfied: gast==0.3.3 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (0.3.3)
Requirement already satisfied: protobuf>=3.9.2 in /software/ml/protobuf-python/3.10.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (3.10.0)
Requirement already satisfied: wheel>=0.26 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (0.37.1)
Requirement already satisfied: astunparse~=1.6.3 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.6.3)
Requirement already satisfied: tensorboard~=2.4 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (2.4.1)
Requirement already satisfied: setuptools in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from protobuf>=3.9.2->tensorflow>=2.4.0->calamari-ocr) (62.1.0)
Requirement already satisfied: markdown>=2.6.8 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (3.3.3)
Requirement already satisfied: werkzeug>=0.11.15 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (1.0.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (
0.4.2)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (1.8
.0)
Requirement already satisfied: google-auth<2,>=1.6.3 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (1.24.0)
Requirement already satisfied: colorama>=0.4.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from adabelief-tf->tfaip==1.2.6->calamari-ocr) (0.4.4)
Requirement already satisfied: tabulate>=0.7 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from adabelief-tf->tfaip==1.2.6->calamari-ocr) (0.8.3)
Requirement already satisfied: gitdb<5,>=4.0.1 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from GitPython->tfaip==1.2.6->calamari-ocr) (4.0.9)
Requirement already satisfied: et-xmlfile in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from openpyxl->tfaip==1.2.6->calamari-ocr) (1.1.0)
Requirement already satisfied: python-dateutil>=2.6.1 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from pandas->tfaip==1.2.6->calamari-ocr) (2.8.0)
Requirement already satisfied: pytz>=2017.2 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from pandas->tfaip==1.2.6->calamari-ocr) (2019.2)
Requirement already satisfied: importlib-metadata in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from prettytable->tfaip==1.2.6->calamari-ocr) (0.22)
Requirement already satisfied: wcwidth in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from prettytable->tfaip==1.2.6->calamari-ocr) (0.1.7)
Requirement already satisfied: networkx>=2.0 in /software/ml/scikit-image/0.16.2-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from scikit-image->tfaip==1.2.6->calamari-ocr) (2.4)
Requirement already satisfied: PyWavelets>=0.4.0 in /software/ml/scikit-image/0.16.2-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from scikit-image->tfaip==1.2.6->calamari-ocr) (1.1.1)
Requirement already satisfied: smmap<6,>=3.0.1 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from gitdb<5,>=4.0.1->GitPython->tfaip==1.2.6->calamari-ocr) (5.0.0)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow>=2.4.0->ca
lamari-ocr) (4.2.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (4.7)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (0.2.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (1.3.0)
Requirement already satisfied: decorator>=4.3.0 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from networkx>=2.0->scikit-image->tfaip==1.2.6->calamari-ocr) (4.4.0)
Requirement already satisfied: idna<2.9,>=2.5 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from requests->pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from requests->pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (2019.9.11)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from requests->pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from requests->pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (1.25.3)
Requirement already satisfied: mypy-extensions>=0.3.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from typing-inspect>=0.4.0->dataclasses-json->tfaip==1.2.6->calamari-ocr) (0.4.3)
Requirement already satisfied: zipp>=0.5 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from importlib-metadata->prettytable->tfaip==1.2.6->calamari-ocr) (0.6.0)
Requirement already satisfied: attrs in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from packaging->pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (19.1.0)
Requirement already satisfied: pyparsing>=2.0.2 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from packaging->pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (2.4.2)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (0.4.7)
Requirement already satisfied: oauthlib>=3.0.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (3.1.0)
Requirement already satisfied: more-itertools in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from zipp>=0.5->importlib-metadata->prettytable->tfaip==1.2.6->calamari-ocr) (7.2.0)
Installing collected packages: calamari-ocr
Attempting uninstall: calamari-ocr
Found existing installation: calamari-ocr 2.2.1
Uninstalling calamari-ocr-2.2.1:
Successfully uninstalled calamari-ocr-2.2.1
Successfully installed calamari-ocr-2.2.2
(Note: I cannot use Docker in my cluster, only Singularity – but that would not help with the different CPU arch.)
bertsky commented
Ah, found the culprit: It seems that TF has a soft dependency on certain Numpy versions. In my case, downgrading to numpy==1.19.5 for tensorflow==2.4.1 helped avoid the above error.
(I have now worked myself up to a problem with warmstarting... coming up.)
Thanks for your feedback!
bertsky commented
For the sake of completeness:
Here is the SO issue that brought me on the right track.
And this is what I had to do ultimately:
pip install --no-use-pep517 numpy==1.19.5
bertsky commented
Note: the issue re-appeared for me when I tried training models for Calamari 1 again (v1.0.5, everything else as above). This time numpy==1.19.5 did not help. Neither did installing OpenCV 4.5.5 (instead of 4.2.0).
Meanwhile, I found this TF issue, which seems to explain what we see, but offers no solution to date – only tf.compat.v1.disable_eager_execution() as a workaround.
bertsky commented
FYI, I just tried that in a brutish way, by adding that line to calamari_ocr.ocr.backends.tensorflow_backend right at the top at module level.
That made the error…
TypeError: Keras symbolic inputs/outputs do not implement `op`. You may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.
…go away, but now I get…
NotImplementedError: Cannot convert a symbolic Tensor (ReadVariableOp_32:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
Something's wrong in the state of Denmark here.