Path to reader version 3, writer version 7 transaction log protocol support

Question

Path to reader version 3, writer version 7 transaction log protocol support

MrPowers opened this issue a year ago · 9 comments

Description

Delta Lake transitioned to table features and now has a better upgrade path for connectors.

delta-rs can upgrade to protocol version 3,7 (skipping the intermediate protocol versions) and then cherry pick the table features that will be supported.

Adding table feature support may also make it safer when reading Delta tables with delta-rs. Does delta-rs gracefully error out when reading a table with table features that are enabled that delta-rs does not currently support?

The purpose of this issue is to organize the remaining work that's needed to get protocol version 3,7 support. Seems like delta-rs is close.

Answer 1 · 2023-09-21T18:00:11.000Z

Does delta-rs gracefully error out when reading a table with table features that are enabled that delta-rs does not currently support?

This should happen right now, so if anyone is getting incorrect data back from delta-rs, we should consider that a bug.

Answer 2 · 2023-09-21T18:00:40.000Z

The purpose of this issue is to organize the remaining work that's needed to get protocol version 3,7 support. Seems like delta-rs is close.

I think the tentative plan is to wait to the delta-kernel-rs to be released publicly and for us to switch that over.

Answer 3 · 2023-11-16T17:18:28.000Z

Once this is merged, we should have full support for append only tables.

Answer 4 · 2023-11-16T17:26:03.000Z

@MrPowers does that mean the rust writer automatically checks it?

Answer 5 · 2023-11-16T17:53:11.000Z

does that mean the rust writer automatically checks it?

This check is done in the commit function, since we are also committing actions to the protocol via that function it should cover all code paths, including writing via python.

Answer 6 · 2023-11-16T18:01:28.000Z

does that mean the rust writer automatically checks it?

This check is done in the commit function, since we are also committing actions to the protocol via that function it should cover all code paths, including writing via python.

Great, then I can remove the delta append only check for the python bindings to rust writer once this is in

Answer 7 · 2023-11-18T17:30:22.000Z

Okay so I have to check the write paths are respected for invariants and fix that if necessary to ensure we have invariant support throughout. Then once that is done I'll start implementing check constraints.

Answer 8 · 2024-01-02T15:32:02.000Z

@hntd187 - can we claim reader version 3, writer version 7 transaction log protocol support now?

Answer 9 · 2024-01-02T18:41:56.000Z

I believe what I signed up to do is done and dusted, so barring any other support we need to support it, I believe so.