Python3 JSON Adapter byte string is not serializable
Amorik opened this issue · 0 comments
Amorik commented
Use of the python3 branch and the JSON adapter results in the following error:
Process Process-3:
Traceback (most recent call last):
File "/opt/eNcore/estreamer/baseproc.py", line 208, in receiveInput
self.onReceive( item )
File "/opt/eNcore/estreamer/baseproc.py", line 313, in onReceive
self.onEvent( item )
File "/opt/eNcore/estreamer/pipeline.py", line 397, in onEvent
data = transform( item, self.settings )
File "/opt/eNcore/estreamer/pipeline.py", line 205, in transform
output = adapters[ index ].dumps( event['record'] )
File "/opt/eNcore/estreamer/adapters/json.py", line 30, in dumps
return json.dumps( data )
File "/usr/local/lib/python3.7/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.7/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable
Im not very familiar with all of the code yet but it appears that the dictionary data being dumped by json.dumps contains values of byte/bytestrings/bytesequences. This was fine in python2 since it handled the encoding/decoding but that not longer happens on its own in the python3 version of JSON.
To resolve i switched from import json to import simplejson as simplejson can handle the bytes conversion. This might not be the right approach for this project, but either way the bytes objects needs to be UTF8 encoded to be JSON compliant/serializable.