delta-io/connectors

cluster flink 1.14.6, Which version of delta-flink should be selected

myBigbug opened this issue · 2 comments

myBigbug commented

cluster flink 1.14.6, Which version of delta-flink should be selected?

0.5.0 or 0.6.0 ?

kristoffSC commented

Hi @myBigbug
Flink 1.14.x was never fully supported for released delta-flink versions.

0.5.0 will work only with Flink 1.13.x
0.6.0 will work with Flink 1.15.x and 1.16.x

Due to few Flink bugs that we have fixed which were causing data loss, we would recommend using Flink >= 1.15.3 or Flink >= 1.16.1. Funny enough, Flink 1.16.0 does not have those fixes.
The Flink 1.14 also had issues with data loss for Sink architecture [2] but since 1.14 has no community support anymore, that one was never fixed.

Regarding 1.14.x, there was a time when connector SNAPSHOT version was prepared to work with 1.14.x [1] but shortly after that, we have switched to 1.15. Flink 1.15 introduced features like writing complex types for Parquet format [3] so from Delta connector point of view it had more sense to use 1.15.

Summarizing Flink 1.14 problems:

  • data loss issue in Flink 1.14,
  • not supported by Flink community anymore (community supports last two minor versions, currently 1.15 and 1.16),
  • unable to write complex Parquet types.

I would recommend migration to Flink 1.15.4 or 1.16.1.

[1] https://github.com/delta-io/connectors/pull/427/files
[2] https://issues.apache.org/jira/browse/FLINK-29589
[3] https://issues.apache.org/jira/browse/FLINK-17782