engine flush的时候偶尔会出现文件找不到的异常
Closed this issue · 1 comments
weizijun commented
报如下错误，需处理listAllHavenaskFiles抛出的异常
[2024-01-16T17:58:51,123][WARN ][o.e.i.s.IndexShard ] [es-cn-jaj3h0sii000cu2eh-data-i-2] [logs-231998][11] failed to flush index
org.elasticsearch.index.engine.FlushFailedEngineException: Flush failed
at org.elasticsearch.index.engine.InternalEngine.refreshLastCommittedSegmentInfos(InternalEngine.java:2009) ~[elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:1967) ~[elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:1205) ~[elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.index.shard.IndexShard$6.doRun(IndexShard.java:3319) [elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) [elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.0.jar:7.10.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:979) [?:?]
Caused by: org.apache.lucene.index.CorruptIndexException: Hit unexpected exception while reading segment infos (resource=commit(null))
at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:220) ~[elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.index.store.Store.readLastCommittedSegmentsInfo(Store.java:198) ~[elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.index.engine.InternalEngine.refreshLastCommittedSegmentInfos(InternalEngine.java:2000) ~[elasticsearch-7.10.0.jar:7.10.0]
... 8 more
Caused by: java.lang.RuntimeException: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/havenask/runtimedata/logs-231998/generation_0/partition_40051_43691/__TASK__/task_1705399111716929/__FENCE__1705399111716929/6/__relocatable__6
at org.havenask.engine.index.store.HavenaskDirectory.lambda$listAllHavenaskFiles$1(HavenaskDirectory.java:62) ~[?:?]
at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listAllHavenaskFiles(HavenaskDirectory.java:57) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.lambda$listAllHavenaskFiles$1(HavenaskDirectory.java:60) ~[?:?]
at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listAllHavenaskFiles(HavenaskDirectory.java:57) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.lambda$listAllHavenaskFiles$1(HavenaskDirectory.java:60) ~[?:?]
at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listAllHavenaskFiles(HavenaskDirectory.java:57) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.lambda$listAllHavenaskFiles$1(HavenaskDirectory.java:60) ~[?:?]
at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listAllHavenaskFiles(HavenaskDirectory.java:57) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.lambda$listAllHavenaskFiles$1(HavenaskDirectory.java:60) ~[?:?]
at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listAllHavenaskFiles(HavenaskDirectory.java:57) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listHavenaskFiles(HavenaskDirectory.java:44) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listAll(HavenaskDirectory.java:38) ~[?:?]
at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:64) ~[lucene-core-8.7.0-ali1.9.8.jar:8.7.0-SNAPSHOT 4a7b45400be872a911e0f74792a3f7a44b8440a2 - admin - 2023-10-20 16:25:09]
at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:64) ~[lucene-core-8.7.0-ali1.9.8.jar:8.7.0-SNAPSHOT 4a7b45400be872a911e0f74792a3f7a44b8440a2 - admin - 2023-10-20 16:25:09]
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:711) ~[lucene-core-8.7.0-ali1.9.8.jar:8.7.0-SNAPSHOT 4a7b45400be872a911e0f74792a3f7a44b8440a2 - admin - 2023-10-20 16:25:09]
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683) ~[lucene-core-8.7.0-ali1.9.8.jar:8.7.0-SNAPSHOT 4a7b45400be872a911e0f74792a3f7a44b8440a2 - admin - 2023-10-20 16:25:09]
at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:474) ~[lucene-core-8.7.0-ali1.9.8.jar:8.7.0-SNAPSHOT 4a7b45400be872a911e0f74792a3f7a44b8440a2 - admin - 2023-10-20 16:25:09]
at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:141) ~[elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:213) ~[elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.index.store.Store.readLastCommittedSegmentsInfo(Store.java:198) ~[elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.index.engine.InternalEngine.refreshLastCommittedSegmentInfos(InternalEngine.java:2000) ~[elasticsearch-7.10.0.jar:7.10.0]
... 8 more
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/havenask/runtimedata/logs-231998/generation_0/partition_40051_43691/__TASK__/task_1705399111716929/__FENCE__1705399111716929/6/__relocatable__6
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:432) ~[?:?]
at java.nio.file.Files.newDirectoryStream(Files.java:472) ~[?:?]
at java.nio.file.Files.list(Files.java:3699) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listAllHavenaskFiles(HavenaskDirectory.java:56) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.lambda$listAllHavenaskFiles$1(HavenaskDirectory.java:60) ~[?:?]
at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listAllHavenaskFiles(HavenaskDirectory.java:57) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.lambda$listAllHavenaskFiles$1(HavenaskDirectory.java:60) ~[?:?]
at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listAllHavenaskFiles(HavenaskDirectory.java:57) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.lambda$listAllHavenaskFiles$1(HavenaskDirectory.java:60) ~[?:?]
at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listAllHavenaskFiles(HavenaskDirectory.java:57) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.lambda$listAllHavenaskFiles$1(HavenaskDirectory.java:60) ~[?:?]
at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listAllHavenaskFiles(HavenaskDirectory.java:57) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.lambda$listAllHavenaskFiles$1(HavenaskDirectory.java:60) ~[?:?]
at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listAllHavenaskFiles(HavenaskDirectory.java:57) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listHavenaskFiles(HavenaskDirectory.java:44) ~[?:?]
at org.havenask.engine.index.store.HavenaskDirectory.listAll(HavenaskDirectory.java:38) ~[?:?]
at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:64) ~[lucene-core-8.7.0-ali1.9.8.jar:8.7.0-SNAPSHOT 4a7b45400be872a911e0f74792a3f7a44b8440a2 - admin - 2023-10-20 16:25:09]
at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:64) ~[lucene-core-8.7.0-ali1.9.8.jar:8.7.0-SNAPSHOT 4a7b45400be872a911e0f74792a3f7a44b8440a2 - admin - 2023-10-20 16:25:09]
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:711) ~[lucene-core-8.7.0-ali1.9.8.jar:8.7.0-SNAPSHOT 4a7b45400be872a911e0f74792a3f7a44b8440a2 - admin - 2023-10-20 16:25:09]
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683) ~[lucene-core-8.7.0-ali1.9.8.jar:8.7.0-SNAPSHOT 4a7b45400be872a911e0f74792a3f7a44b8440a2 - admin - 2023-10-20 16:25:09]
at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:474) ~[lucene-core-8.7.0-ali1.9.8.jar:8.7.0-SNAPSHOT 4a7b45400be872a911e0f74792a3f7a44b8440a2 - admin - 2023-10-20 16:25:09]
at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:141) ~[elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:213) ~[elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.index.store.Store.readLastCommittedSegmentsInfo(Store.java:198) ~[elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.index.engine.InternalEngine.refreshLastCommittedSegmentInfos(InternalEngine.java:2000) ~[elasticsearch-7.10.0.jar:7.10.0]
... 8 more