Issue in ingesting nested dictionaries in a Python South plugin

[sudhirkshirsagar]

Sep 24 21:27:07 sudhir-Aspire-5749 Fledge mqn1[13257] WARNING: sending_process: sending_process_mqn1: unable to convert in memory data structure related to the readings data - error details |float() argument must be a string or a number, not 'dict'| - row |{'id': 637, 'asset_code': 'sample', 'reading': {'name': 'A', 'foo': {'nest': 5}}, 'user_ts': '2020-09-25 04:27:03.899686', 'ts': '2020-09-25 04:27:04.396'}|

[sudhirkshirsagar]

The readings appear in the dashboard but with the nested dictionary as an object. So the ingest may be OK but the processing may be an issue.

[sudhirkshirsagar]

I will try it with http north

[sudhirkshirsagar]

I get the same error with the http_north plugin. I will open this there.

[sudhirkshirsagar]

I was able to use the httpc successfully with the nested json. Looking at the error a bit more, it is definitely from Fledge python code and not the north plugins.
{'id': 637, 'asset_code': 'sample', 'reading': {'name': 'A', 'foo': {'nest': 5}}, 'user_ts': '2020-09-25 04:27:03.899686', 'ts': '2020-09-25 04:27:04.396'}
The north plugin don't have variables like id or asset_code.

[MarkRiddoch]

Thanks for this, we do have an issue with some destinations that they can not support nested objects and require that we flatten the structure, however it looks like you have found a couple of places where we do not correctly handle nesting. I will get this added to the project task list and looked into.

[sudhirkshirsagar]

Thanks Mark. I got around by hacking your GCP code. I like your coding style!

[ashish-jabble]

@MarkRiddoch We may close this out as we have support the nested objects for python north tasks in develop branch via PR #232

[MarkRiddoch]

A fix for this will be included in the 1.9 release of Fledge that is currently under preparation.

[MarkRiddoch]

1.9 is now released and packages are available.