delta-io/connectors

Flink Complex Data Types - Support for Flink v1.15.x

adriandavidsmith opened this issue · 2 comments

adriandavidsmith commented

Using Flink v1.14.5 and the Respective Flink Delta Connector v0.4.1, it appears that complex data types are not supported when writing data to S3.

i.e.

...
val ROW_TYPE = RowType(
    listOf(
        ...
        RowType.RowField("foo", VarCharType(VarCharType.MAX_LENGTH)),
        RowType.RowField("bar", ArrayType(VarCharType())),
        ...
    )
)
...

yields the following

Caused by: java.lang.UnsupportedOperationException: Unsupported type: ARRAY<VARCHAR(1)>
	at org.apache.flink.formats.parquet.utils.ParquetSchemaConverter.convertToParquetType(ParquetSchemaConverter.java:105)
	at org.apache.flink.formats.parquet.utils.ParquetSchemaConverter.convertToParquetType(ParquetSchemaConverter.java:43)
        ...

This seems to be related to Flink not being able to do these conversions. Upon further investigations, this issue seems to have been fixed as part of Flink 1.15.x in following ticket and the following pull request respectfully.

Looking at documentation for the connectors, there are currently no versions of the Connectors for Flink available for Flink v1.15.x.

Looking at the roadmap for H2 of 2022, it does not appear that the support for Flink 1.15.x is on the agenda.

Is there something that I may have missed as part of the documentation around writing complex data types using the Flink Delta Connector(s) which would support writing complex data types (such as arrays) using any version of the Delta Connector(s) and any version of Flink?

If the above is correct, is there still capacity to include this support in the H2 roadmap?

kristoffSC commented

Hi,
thanks for the feedback.

Updating to Flink 1.15 is actually under evaluation for "Flink SQL+ Table API + Catalog Support" item from the road map. We are considering 1.14 vs 1.15 for this item.

Additionally please note that Flink 1.15 supports only writes for composite types, Reads are still unavailable in that version.
This is important information since newest connectors release, version 0.5.0 includes Flink Delta Source

  • 👍3