starburstdata/dbt-trino

Support `CREATE OR REPLACE`

Fokko opened this issue · 2 comments

Fokko commented

Describe the feature

Hey everyone!

I'm delighted to see CREATE OR REPLACE support in Trino: trinodb/trino#13180

This will both:

  • History of the table will be preserved in the Iceberg metadata, making it more transparent to the user what happened to the table.
  • Make the execution of the pipeline faster, since fewer statements are being issued

Describe alternatives you've considered

I would love to work on this, but my main question is how to determine if this is supported. I checked, but didn't find any minimal version to enable certain features.

Who will benefit?

Everyone using Iceberg in Trino 🚀

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
hovaesco commented

I'm not aware of anyone working on this, let's introduce it as a separate strategy configured as on_table_exists = 'replace'.

I don't think we have a way to determine if this is supported on server side, please correct me if I'm wrong @mdesmet

  • 👍1
mosabua commented

Theoretically you could get the version of the server (via various means including version() function) and decide with that info. The problem is that it will be different for each connector so it will only be true after you figure out the connector used in a specific catalog (which you can also do via a query).

You could just leave it to the user to worry about and deal with it in the same way you deal with any other statement that might or might not be supported by the Trino version of the server and the connector. For example think of MERGE or TRUNCATE .. are you doing anything about these in terms of validation?