Support for reading and writing complex nested types
kristoffSC opened this issue · 4 comments
Currently Delta Sink and Source can read and write complex types such as List, Map, Rows ect.
However due to current limitations of Flink (1.16), those complex types can be used only with primitive types, for example List of Strings,
List of Integers etc.
Currently Complex nested types are not supported, which means that reading or writing Schema such as
Row<Array<Array>> or Row<Array>> is not supported.
Appropriate Flink tickets were created to cover this problem:
https://issues.apache.org/jira/browse/FLINK-31197
https://issues.apache.org/jira/browse/FLINK-31202
@kristoffSC any update on this? Doesn't seem like any updates on the linked tickets.
When trying to write complex nested RowData using the Delta Sink I get the following error:
Caused by: java.lang.RuntimeException: org.apache.parquet.io.ParquetEncodingException: empty fields are illegal, the field should be ommited completely instead .
No updates from my end.
Issue comes from Flink, so someone has to contribute there :)
This repo has been deprecated and the code is moved under connectors module in https://github.com/delta-io/delta repository. Please create the issue in repository https://github.com/delta-io/delta. See #556 for details.