Training Fails: Not JSON Serializable
lmangani opened this issue · 1 comments
lmangani commented
Hello @regel & All,
Testing release 1.6.0 after a long time (installed via yum packages, managed through the grafana integration) and I noticed any successful training consistently fails being stored at the process tail with the following error, apparently regardless of the bucket settings/range:
127.0.0.1 - - [2020-10-24 15:11:25] "GET /models/hepic_hepic_statistics_calls_mean_finished__time_5m HTTP/1.1" 200 701 0.002114
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [01:00<00:00, 6.01s/it, best loss: 18.589042263891084]
ERROR:root:('Not JSON Serializable:', b'\n\x11SquaredDifference\x12\x11SquaredDifference\x1a\x17decoder_dense_2/BiasAdd\x1a\x07input_1*\x07\n\x01T\x12\x020\x01')
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/loudml-1.6.0-py3.6.egg/loudml/worker.py", line 53, in run
res = getattr(self, func_name)(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/loudml-1.6.0-py3.6.egg/loudml/worker.py", line 101, in train
**kwargs
File "/usr/local/lib/python3.6/site-packages/loudml-1.6.0-py3.6.egg/loudml/donut.py", line 1101, in train
model_b64 = _serialize_keras_model(self._keras_model)
File "/usr/local/lib/python3.6/site-packages/loudml-1.6.0-py3.6.egg/loudml/donut.py", line 228, in _serialize_keras_model
keras_model.save(path)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1211, in save
saving.save_model(self, filepath, overwrite, include_optimizer, save_format)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/keras/saving/save.py", line 113, in save_model
model, filepath, overwrite, include_optimizer)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 101, in save_model_to_hdf5
default=serialization.get_json_type).encode('utf8')
File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/util/serialization.py", line 69, in get_json_type
raise TypeError('Not JSON Serializable:', obj)
TypeError: ('Not JSON Serializable:', b'\n\x11SquaredDifference\x12\x11SquaredDifference\x1a\x17decoder_dense_2/BiasAdd\x1a\x07input_1*\x07\n\x01T\x12\x020\x01')
ERROR:root:job[469672de-928b-4424-b5bc-3935fdd665ad] failed: ('Not JSON Serializable:', b'\n\x11SquaredDifference\x12\x11SquaredDifference\x1a\x17decoder_dense_2/BiasAdd\x1a\x07input_1*\x07\n\x01T\x12\x020\x01')
Did anyone else experience the same and/or has a clue as of what might be causing this?
Thanks in advance!
lmangani commented
Add-on: This seems to only affect the RPM package. Using the Docker container, everything runs smoothly.