memiiso/debezium-server-iceberg

Number type of Oracle don't right with data type of Iceberg table

Opened this issue · 3 comments

duc-dn commented

  • I am ingesting data from oracle to iceberg table. This is data types of source table in my oracle
    image

  • And this is data types in iceberg table.
    image

  • Data in iceberg table
    image

  • I see that number datatype in oracle don't right with data type of iceberg table

ismailsimsek commented

@duc-dn by default NUMBER is mapped by debezium. you could use decimal.handling.mode debezium configuration to enforce different handling.

duc-dn commented

Thanks @ismailsimsek, I added "debezium.source.decimal.handling.mode": "double" and resolved this problem

  • 👍1
duc-dn commented

Hi @ismailsimsek, I tried to cdc with a higher version of Oracle (19) and got the error below. After I ran with Postgres, MySql, I got the same error. Before that, I was still able to run cdc with Postgres, MySql without any problem and I didn't change anything about my docker image.

2023-10-28 13:46:28,486 ERROR [io.deb.ser.ConnectorLifecycle] (pool-7-thread-1) Connector completed: success = 'false', message = 'Stopping connector after error in the application's handler method: 'org.apache.iceberg.shaded.com.fasterxml.jackson.databind.ObjectMapper org.apache.iceberg.util.JsonUtil.mapper()'', error = 'java.lang.NoSuchMethodError: 'org.apache.iceberg.shaded.com.fasterxml.jackson.databind.ObjectMapper org.apache.iceberg.util.JsonUtil.mapper()'': java.lang.NoSuchMethodError: 'org.apache.iceberg.shaded.com.fasterxml.jackson.databind.ObjectMapper org.apache.iceberg.util.JsonUtil.mapper()'
at org.apache.iceberg.hive.HiveTableOperations.setSnapshotSummary(HiveTableOperations.java:470)
at org.apache.iceberg.hive.HiveTableOperations.setSnapshotStats(HiveTableOperations.java:461)
at org.apache.iceberg.hive.HiveTableOperations.setHmsTableParameters(HiveTableOperations.java:440)
at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:237)
at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:400)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:374)
at io.debezium.server.iceberg.tableoperator.IcebergTableOperator.addToTablePerSchema(IcebergTableOperator.java:182)
at io.debezium.server.iceberg.tableoperator.IcebergTableOperator.addToTable(IcebergTableOperator.java:157)
at io.debezium.server.iceberg.IcebergChangeConsumer.handleBatch(IcebergChangeConsumer.java:167)
at io.debezium.embedded.ConvertingEngineBuilder.lambda$notifying$2(ConvertingEngineBuilder.java:101)
at io.debezium.embedded.EmbeddedEngine.pollRecords(EmbeddedEngine.java:1008)
at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:755)
at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:229)
at io.debezium.server.DebeziumServer.lambda$start$1(DebeziumServer.java:170)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2023-10-28 13:46:28,501 INFO [io.deb.ser.DebeziumServer] (main) Received request to stop the engine
2023-10-28 13:46:28,501 INFO [io.deb.emb.EmbeddedEngine] (main) Stopping the embedded engine
2023-10-28 13:46:28,546 INFO [io.quarkus] (main) debezium-server-iceberg-dist stopped in 0.058s

image