[Bug]: Fix `File format is required` when scanning file entries if no file format is specified in the file name
Closed this issue · 0 comments
wangtaohz commented
What happened?
When there is no file format indicated in the file name(like xxxx.parquet
, yyyy.avro
), TableEntriesScan will throw an error File format is required.
Affects Versions
master
What engines are you seeing the problem on?
Core
How to reproduce
No response
Relevant log output
[com.netease.arctic.server.table.executor.SnapshotsExpiringExecutor] [] - unexpected expire error of table arcti
c_xxx.yyy.zzz(tableId=155)
java.lang.IllegalArgumentException: File format is required
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:145) ~[iceberg-bundled-guava-1.3.0.jar:?]
at org.apache.iceberg.DataFiles$Builder.build(DataFiles.java:293) ~[iceberg-core-1.3.0.jar:?]
at com.netease.arctic.scan.TableEntriesScan.buildDataFile(TableEntriesScan.java:325) ~[amoro-core-0.6.0-SNAPSHOT.jar:?]
at com.netease.arctic.scan.TableEntriesScan.buildContentFile(TableEntriesScan.java:301) ~[amoro-core-0.6.0-SNAPSHOT.jar:?]
at com.netease.arctic.scan.TableEntriesScan.lambda$entries$0(TableEntriesScan.java:240) ~[amoro-core-0.6.0-SNAPSHOT.jar:?]
at org.apache.iceberg.io.CloseableIterable$7$1.next(CloseableIterable.java:202) ~[iceberg-api-1.3.0.jar:?]
at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:65) ~[iceberg-api-1.3.0.jar:?]
at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:49) ~[iceberg-api-1.3.0.jar:?]
at com.netease.arctic.server.utils.IcebergTableUtil.getAllContentFilePath(IcebergTableUtil.java:155) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
at com.netease.arctic.server.optimizing.maintainer.MixedTableMaintainer.<init>(MixedTableMaintainer.java:91) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
at com.netease.arctic.server.optimizing.maintainer.TableMaintainer.ofTable(TableMaintainer.java:46) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
at com.netease.arctic.server.table.executor.SnapshotsExpiringExecutor.execute(SnapshotsExpiringExecutor.java:59) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
at com.netease.arctic.server.table.executor.BaseTableExecutor.executeTask(BaseTableExecutor.java:72) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
at com.netease.arctic.server.table.executor.BaseTableExecutor.lambda$scheduleIfNecessary$4(BaseTableExecutor.java:81) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_101]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
Anything else
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct