apache/amoro

[Bug]: Fix `File format is required` when scanning file entries if no file format is specified in the file name

Closed this issue · 0 comments

What happened?

When there is no file format indicated in the file name(like xxxx.parquet, yyyy.avro), TableEntriesScan will throw an error File format is required.

Affects Versions

master

What engines are you seeing the problem on?

Core

How to reproduce

No response

Relevant log output

[com.netease.arctic.server.table.executor.SnapshotsExpiringExecutor] [] - unexpected expire error of table arcti
c_xxx.yyy.zzz(tableId=155)
java.lang.IllegalArgumentException: File format is required
        at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:145) ~[iceberg-bundled-guava-1.3.0.jar:?]
        at org.apache.iceberg.DataFiles$Builder.build(DataFiles.java:293) ~[iceberg-core-1.3.0.jar:?]
        at com.netease.arctic.scan.TableEntriesScan.buildDataFile(TableEntriesScan.java:325) ~[amoro-core-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.scan.TableEntriesScan.buildContentFile(TableEntriesScan.java:301) ~[amoro-core-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.scan.TableEntriesScan.lambda$entries$0(TableEntriesScan.java:240) ~[amoro-core-0.6.0-SNAPSHOT.jar:?]
        at org.apache.iceberg.io.CloseableIterable$7$1.next(CloseableIterable.java:202) ~[iceberg-api-1.3.0.jar:?]
        at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:65) ~[iceberg-api-1.3.0.jar:?]
        at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:49) ~[iceberg-api-1.3.0.jar:?]
        at com.netease.arctic.server.utils.IcebergTableUtil.getAllContentFilePath(IcebergTableUtil.java:155) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.server.optimizing.maintainer.MixedTableMaintainer.<init>(MixedTableMaintainer.java:91) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.server.optimizing.maintainer.TableMaintainer.ofTable(TableMaintainer.java:46) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.server.table.executor.SnapshotsExpiringExecutor.execute(SnapshotsExpiringExecutor.java:59) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.server.table.executor.BaseTableExecutor.executeTask(BaseTableExecutor.java:72) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.server.table.executor.BaseTableExecutor.lambda$scheduleIfNecessary$4(BaseTableExecutor.java:81) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_101]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_101]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]

Anything else

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

  • I agree to follow this project's Code of Conduct