Payload too big to get parsed
Closed this issue · 1 comments
Having connected multiple buttons to my FlicHub, I get an error, that the JSON cannot be parsed due to varying errors. Looking at the logs one can see, that the payload for the buttons gets split up in multiple sections, which results in an error, as the payload gets split up somewhere within the JSON string, which than cannot be parsed correctly due to missing content afterwards.
Limiting the payload to send data from only 3-4 buttons (depending on whether they are V1 or V2 buttons), the payload is small enought to not be split up and therefore gets parsed correctly and therefore also shows the different buttons as valid entities within HomeAssistant. Is there a fix for that?
Here is some example output of the sent payload which gets cut off and continued in a second payload after a certain amount of letters:
2024-02-21 13:52:37.876 WARNING (MainThread) [pyflichub.client] Unable to decode received data 2024-02-21 13:52:37.885 DEBUG (MainThread) [pyflichub.client] Data received: '{"command":"buttons","data":[{"bdaddr":"80:e4:da:76:46:3e","serialNumber":"BA33-A01193","color":"","name":"Bed","activeDisconnect":false,"connected":true,"ready":true,"batteryStatus":38,"batteryTimestamp":1708519179964,"uuid":"ea739305bbcd410380cb591232e623ee","flicVersion":2,"firmwareVersion":11,"key":"2bd802b66ef327243a2bb7bd7958d298d5ae3b79","passiveMode":false},{"bdaddr":"80:e4:da:76:57:88","serialNumber":"BA33-A05647","color":"","name":"AC","activeDisconnect":false,"connected":false,"ready":false,"batteryStatus":100,"batteryTimestamp":1694058647178,"uuid":"01779e107e66437fa400a86f2d1c2f2f","flicVersion":2,"firmwareVersion":11,"key":"12a3bbd63d89a62b9835d94a51c78f3ba1b1abae","passiveMode":false},{"bdaddr":"80:e4:da:76:66:05","serialNumber":"BA33-A09367","color":"","name":"TV","activeDisconnect":false,"connected":true,"ready":true,"batteryStatus":40,"batteryTimestamp":1708512441173,"uuid":"febbed83210b4ad5a5e86fbbab519203","flicVersion":2,"firmwareVersion":11,"key":"2f4cce8cbd5a6ee9988c17d0920ba02bc44ce5b9","passiveMode":false},{"bdaddr":"80:e4:da:76:98:1b","serialNumber":"BA46-A22457","color":"","name":"Entrance Bed/Living Room","activeDisconnect":false,"connected":true,"ready":true,"batteryStatus":38,"batteryTimestamp":1708519191663,"uuid":"16c5ed02f74f4611909efde20db34a0a","flicVersion":2,"firmwareVersion":11,"key":"0ed2bcbdcab92ccc64ae20e203348d8bd87acb71","passiveMode":false},{"bdaddr":"80:e4:da:76:9d:09","serialNumb' 2024-02-21 13:52:37.885 WARNING (MainThread) [pyflichub.client] Unterminated string starting at: line 1 column 1438 (char 1437) Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/pyflichub/client.py", line 148, in data_received msg = json.loads(data_part, cls=_JSONDecoder) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/json/__init__.py", line 359, in loads return cls(**kw).decode(s) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) ^^^^^^^^^^^^^^^^^^^^^^ json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1438 (char 1437) 2024-02-21 13:52:37.887 WARNING (MainThread) [pyflichub.client] Unable to decode received data 2024-02-21 13:52:37.894 DEBUG (MainThread) [pyflichub.client] Data received: 'er":"BA46-A23381","color":"","name":"Musik","activeDisconnect":false,"connected":false,"ready":false,"batteryStatus":47,"batteryTimestamp":1691341967929,"uuid":"3d11fcb02cdf4bc79740b9ac55996452","flicVersion":2,"firmwareVersion":10,"key":"76410d9aed0f00932ab3ee3e6bcbb3a89cfab40e","passiveMode":false}]}\n' 2024-02-21 13:52:37.894 WARNING (MainThread) [pyflichub.client] Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/pyflichub/client.py", line 148, in data_received msg = json.loads(data_part, cls=_JSONDecoder) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/json/__init__.py", line 359, in loads return cls(**kw).decode(s) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 2024-02-21 13:52:37.897 WARNING (MainThread) [pyflichub.client] Unable to decode received data
I'm seeing the same here; I have a total of 16 buttons registered in the hub and the payload gets split into two and each half causes an error. None of the button names have anything other than alphanumeric and space characters, and the split happens in the middle of a non-button-name element (namely the batteryTimestamp key).
Here's a log excerpt (with redacted button names):
2024-05-11 09:23:19.577 DEBUG (MainThread) [pyflichub.client] Data received: '{"command":"buttons","data":[{"bdaddr":"80:e4:da:74:b8:54","serialNumber":"AH30-D03506","color":"white","name":"redacted1","activeDisconnect":false,"connected":true,"ready":true,"batteryStatus":27,"batteryTimestamp":1695753484954,"uuid":"bafcf4f265cc4168b8b60a101cd6af51","flicVersion":1,"firmwareVersion":0,"key":null,"passiveMode":false},{"bdaddr":"80:e4:da:74:b8:a5","serialNumber":"AH30-D04607","color":"black","name":"redacted2","activeDisconnect":false,"connected":false,"ready":false,"batteryStatus":0,"batteryTimestamp":1662124804003,"uuid":"9a97dda9d65e41bcaba80da2d109b44d","flicVersion":1,"firmwareVersion":0,"key":null,"passiveMode":false},{"bdaddr":"80:e4:da:74:b8:bd","serialNumber":"AH30-D04711","color":"white","name":"redacted3","activeDisconnect":false,"connected":false,"ready":false,"batteryStatus":7,"batteryTimestamp":1686892309425,"uuid":"21fbdaffc18f4ea8b185eaf91bf16b71","flicVersion":1,"firmwareVersion":0,"key":null,"passiveMode":false},{"bdaddr":"80:e4:da:74:b8:c0","serialNumber":"AH30-D04705","color":"black","name":"redacted4","activeDisconnect":false,"connected":true,"ready":true,"batteryStatus":6,"batteryTimestamp":1695744619176,"uuid":"c0547afa42074d718e26ba455f78daa9","flicVersion":1,"firmwareVersion":0,"key":null,"passiveMode":false},{"bdaddr":"80:e4:da:75:2d:a5","serialNumber":"AH39-D09311","color":"white","name":"redacted5","activeDisconnect":false,"connected":false,"ready":false,"batteryStatus":23,"batter'
2024-05-11 09:23:19.577 WARNING (MainThread) [pyflichub.client] Unterminated string starting at: line 1 column 1442 (char 1441)
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/pyflichub/client.py", line 148, in data_received
msg = json.loads(data_part, cls=_JSONDecoder)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/json/__init__.py", line 359, in loads
return cls(**kw).decode(s)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1442 (char 1441)
2024-05-11 09:23:19.579 WARNING (MainThread) [pyflichub.client] Unable to decode received data
2024-05-11 09:23:19.580 DEBUG (MainThread) [pyflichub.client] Data received: 'yTimestamp":1654624931592,"uuid":"1c819c33882e45c29829ff1393dfb02f","flicVersion":1,"firmwareVersion":0,"key":null,"passiveMode":false},{"bdaddr":"80:e4:da:75:35:1b","serialNumber":"AH45-D02548","color":"white","name":"redacted6","activeDisconnect":false,"connected":false,"ready":false,"batteryStatus":0,"batteryTimestamp":1656521974366,"uuid":"f42d4adba76945899eb919964b75fe2d","flicVersion":1,"firmwareVersion":0,"key":null,"passiveMode":false},{"bdaddr":"80:e4:da:75:35:1d","serialNumber":"AH45-D02551","color":"black","name":"redacted7","activeDisconnect":false,"connected":false,"ready":false,"batteryStatus":2,"batteryTimestamp":1659737076737,"uuid":"b789e94333f54ae1a7a4f76fc830b7d1","flicVersion":1,"firmwareVersion":0,"key":null,"passiveMode":false},{"bdaddr":"80:e4:da:75:3f:54","serialNumber":"AH39-D04335","color":"black","name":"redacted8","activeDisconnect":false,"connected":false,"ready":false,"batteryStatus":17,"batteryTimestamp":1657768353120,"uuid":"a043a4c7d3634214ac0dc66bb73f86f2","flicVersion":1,"firmwareVersion":0,"key":null,"passiveMode":false},{"bdaddr":"80:e4:da:77:44:aa","serialNumber":"BB31-B66384","color":"","name":"redacted9","activeDisconnect":false,"connected":false,"ready":false,"batteryStatus":100,"batteryTimestamp":1710800530935,"uuid":"213bab3d54034e4082b0399b0629793c","flicVersion":2,"firmwareVersion":11,"key":"5adbb1d86f7c1a5a9e0d226a7f5d3388ac129c76","passiveMode":false},{"bdaddr":"80:e4:da:77:53:dc","serialNumber":"BB31-B70985","color":"","name":"redacted10","activeDisconnect":false,"connected":true,"ready":true,"batteryStatus":100,"batteryTimestamp":1715376752065,"uuid":"e4159839093b479ca7f593fa2f306877","flicVersion":2,"firmwareVersion":11,"key":"1ae15b18f778ec7312bd4d11a693d1cd2c88abef","passiveMode":false},{"bdaddr":"80:e4:da:77:53:e8","serialNumber":"BB31-B70997","color":"","name":"redacted11","activeDisconnect":false,"connected":true,"ready":true,"batteryStatus":100,"batteryTimestamp":1715376862998,"uuid":"49a1b52688cd450595cbe9e1e95c2c7e","flicVersion":2,"firmwareVersion":11,"key":"f20e3d733104a21c00fd910d902c8251b09069ba","passiveMode":false},{"bdaddr":"80:e4:da:79:95:12","serialNumber":"BG14-D32054","color":"","name":"redacted12","activeDisconnect":false,"connected":true,"ready":true,"batteryStatus":100,"batteryTimestamp":1715376861651,"uuid":"e7b842c1577f4a0db345cc2daacd3f70","flicVersion":2,"firmwareVersion":11,"key":"90a27681fa49516e6dfeb2a9de382f1f62b5fcd7","passiveMode":false},{"bdaddr":"80:e4:da:79:95:6f","serialNumber":"BG14-D31963","color":"","name":"redacted13","activeDisconnect":false,"connected":true,"ready":true,"batteryStatus":100,"batteryTimestamp":1715376851412,"uuid":"a83aafcc68734fe3b676b33122fe39fa","flicVersion":2,"firmwareVersion":11,"key":"53bddde53b18a72b65199301890df9a73d5e1989","passiveMode":false},{"bdaddr":"80:e4:da:79:95:7b","serialNumber":"BG14-D31969","color":"","name":"redacted14","activeDisconnect":false,"connected":true,"ready":true,"batteryStatus":100,"batteryTimestamp":1715376769617,"uuid":"4d5db8aadeef45439ca4c5ee86a800c1","flicVersion":2,"firmwareVersion":11,"key":"785ee7a8b39e19c96e25df6adc0addaaeb60fad0","passiveMode":false},{"bdaddr":"80:e4:da:79:99:06","serialNumber":"BG14-D33919","color":"","name":"redacted15","activeDisconnect":false,"connected":true,"ready":true,"batteryStatus":100,"batteryTimestamp":1715376806407,"uuid":"d861a709ccc246009e7d6449ee8b1dd6","flicVersion":2,"firmwareVersion":11,"key":"1f9c4376b7f5ef906c4bd3da02205e93edf8e5ae","passiveMode":false},{"bdaddr":"80:e4:da:79:99:68","serialNumber":"BG14-D32616","color":"","name":"redacted16","activeDisconnect":false,"connected":true,"ready":true,"batteryStatus":100,"batteryTimestamp":1715376921955,"uuid":"6db270dfabe0409693438ff18a7c49e0","flicVersion":2,"firmwareVersion":11,"key":"63b328b44499e9d0661f325b06d1549c2b3b1efe","passiveMode":false}]}\n'
2024-05-11 09:23:19.580 WARNING (MainThread) [pyflichub.client] Expecting value: line 1 column 1 (char 0)
Note that the first "Data received" ends with "batteryStatus":23,"batter' and the second one starts with 'yTimestamp":1654624931592,.
It looks like the two payloads concatenated would be perfectly parseable, it's just the size that is the problem. Interestingly it seems like the second half is much larger than the first - the first segment is 1448 characters, while the second is 3849. I wonder if the split is not happening purely based on size (otherwise the second half should itself have been split as well).
I've also tried connecting directly to the Flic Hub's socket (via telnet) and the data appears to be returned without any splitting. Perhaps python is splitting it internally for some reason?