Upgrade Delta Standalone to support higher Delta protocol versions
allisonport-db opened this issue · 3 comments
Motivation
Currently Delta Standalone only supports reader version 1 and writer version 2 of the Delta Protocol. This means any connectors using the library are blocked from reading or writing to higher protocol tables. Therefore, users that use delta-standalone-based connectors (e.g. Flink, Hive, Presto) cannot also use protocol versions higher than that which delta-standalone supports. We would like to update Delta Standalone to support higher protocol versions to unblock connectors and encourage higher-protocol feature adoption. We also would like to update Delta Standalone to support Table Features (delta-io/delta#1408).
Design
See the design doc here for details https://docs.google.com/document/d/1HtzabGHibLZfu30GwWlloAUEzHVuFbDxFroJxKyMTgE/edit?usp=sharing
Project Plan
[coming soon]
@allisonport-db is there anyway today to write out a minWriter version above 2 from the standalone java client? I am trying to create a table with the java client that can later be read by spark jobs that do support these higher versions.
@the-other-tim-brown there is currently no way to write a table with a higher minWriterVersion. However, tables with minWriterVersion=2 should be able to be read by the spark-client
This repo has been deprecated and the code is moved under connectors module in https://github.com/delta-io/delta repository. Please create the issue in repository https://github.com/delta-io/delta. See #556 for details.