DataSQRL/sqrl

Support writing state tables to Snowflake

Opened this issue · 0 comments

Add this to track external limitations of Snowflake / flink iceberg connector.

Currently snowflake does not support equality or positional deletes but support is planned.

The flink iceberg connector does not support copy-on-write or table maintenance so there currently no path to compact a table. Some people suggest using spark solely to compact a table, which is a possible workaround. To get it to work in snowflake, all iceberg tables should annotated which the iceberg metadata file, and would need to be manually updated as new tables are compacted. https://github.com/orgs/apache/projects/358
https://docs.google.com/document/d/16g3vR18mVBy8jbFaLjf2JwAANuYOmIwr15yDDxovdnA