databricks/iceberg-kafka-connect

MSK Connector with tasks.max > 1 throws an AlreadyExistsException while creating glue table

Closed this issue · 0 comments

We have defined a kafka cluster with 3 partitions count. We referred https://tabular.io/blog/streaming-event-data-to-iceberg-with-kafka-connect/ and decided to consume the data with high speed and for that set tasks.max=3.

Below is the exception we noticed ->
org.apache.iceberg.exceptions.AlreadyExistsException: Cannot commit iceberg.ledgers because its Glue table already exists when trying to create one at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:199) at io.tabular.iceberg.connect.data.IcebergWriterFactory.lambda$autoCreateTable$1(IcebergWriterFactory.java:106)
Could you please suggest any alternate way to tackle this case?