newfront/hitchhikers_guide_to_deltalake_streaming

Show the Interceptor Pattern for Streaming Tables

Opened this issue · 1 comments

Brought up by @AndreGodinho7 in the Delta User's slack channel.

When we are migrating from Table A (call it "last stable") and we need to move to the new "stable" table, while not "interrupting" the streams until we are "ready". If there is a new set of "columns" then those can get picked up by the streaming applications without problems. However, if there is a completely re-written table structure, and we need to use a new "uuid" - then the change is more complex. We want to shoot for less complicated.

  • Consider that there are live customers "streaming data customers"
  • You want to create a "new" table to "replace" the old "tableA"
  • You want to ensure there is a "backup plan" - so you intercept the "table" and "redirect"

This is not a simple established pattern. It is a full "table aliasing" concept.

  • talk about ways to get this to work (with Trino for example - you can "redirect" tables to new locations) - with managed tables and a table UUID that changes with "Delta" - we will need a more "seamless" means to handle the redirection.

  • Include an official writeup (blog post) to go along with the pattern.

Evolution on https://en.wikipedia.org/wiki/Interceptor_pattern for streaming data tables.