TupleToJson should flatten Json output if a spl list contains only one element
chanskw opened this issue · 5 comments
Assuming a tuple has the following type:
type
DataType = tuple<rstring a,list<rstring> b,list<rstring> c>;
If a tuple of this data type contains the following:
Data1: data={a="A", b=["B"], c=["C"]}
The following will get generated by the TupleToJSON operator:
{"a":"A","b":["B"],"c":["C"]}
This is cumbersome to parse by downstream applications. User would like to see the following generated instead:
{"a":"A","b":"B","c":"C"}
Since it's standard valid JSON, why would it be cumbersome to parse? Especially as the dwonstream application has to handle multiple values in the list which requires the same parsing.
This is a requirement I received from talking to an external user. I have to check what they are doing downstream. But they do not like having these optional elements embedded. I think in their case, these elements can have one or zero value.
Seems strange that the SPL schema is a list then rather than a single value.
Just seems this special cases have the potential to make the operator become complicated and thus harder for users to understand.
The customer wants to suppress optional elements in the JSON output if the source value is not present. Since SPL does not support optional values, the workaround is either to use additional attributes (e.g. boolean) to indicate that the value is present or absent, or to use a list, which can have 0 or 1 elements.
From my point of view, this issue asks for a non-automatic JSON schema. Today, the JSON schema is derived from the tuple schema. To solve the customer problem, a customer-provided schema would be helpful. This schema would specify an attribute as optional and how to decide whether the value is present or absent.
Example 1 (boolean): JSON "b" is optional, JSON "b" is present if the SPL boolean isBPresent attribute is true, use SPL b value
Example 2 (list): "b" is optional, "b" is present if the size of the SPL b list equals 1, use SPL b[0] value
Old issue, it was about workaround for optional elements which we have meanwhile in Streams and which are supported in JSON toolkit