prestodb/presto-hadoop-apache2

FileSystem closed exception

Closed this issue · 7 comments

wubiaoi commented 
com.facebook.presto.spi.PrestoException: java.io.IOException: Filesystem closed
        at com.facebook.presto.hive.orc.OrcPageSource.getNextPage(OrcPageSource.java:159)
        at com.facebook.presto.hive.HivePageSource.getNextPage(HivePageSource.java:204)
        at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:248)
        at com.facebook.presto.operator.Driver.processInternal(Driver.java:378)
        at com.facebook.presto.operator.Driver.processFor(Driver.java:301)
        at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:622)
        at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:534)
        at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:670)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.IOException: Filesystem closed
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at com.facebook.presto.hive.orc.OrcPageSource.close(OrcPageSource.java:176)
        at com.facebook.presto.hive.orc.OrcPageSource.getNextPage(OrcPageSource.java:137)
        ... 10 more
Caused by: java.io.IOException: Filesystem closed
        at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:822)
        at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:672)
        at java.io.FilterInputStream.close(FilterInputStream.java:181)
        at com.facebook.presto.hive.orc.HdfsOrcDataSource.close(HdfsOrcDataSource.java:42)
        at com.facebook.presto.orc.OrcRecordReader.close(OrcRecordReader.java:281)
        at com.facebook.presto.hive.orc.OrcPageSource.close(OrcPageSource.java:173)
        ... 11 more
dexchannel commented

same with:

com.facebook.presto.spi.PrestoException: Filesystem closed
        at com.facebook.presto.hive.parquet.HdfsParquetDataSource.buildHdfsParquetDataSource(HdfsParquetDataSource.java:108)
        at com.facebook.presto.hive.parquet.ParquetHiveRecordCursor.createParquetRecordReader(ParquetHiveRecordCursor.java:331)
        at com.facebook.presto.hive.parquet.ParquetHiveRecordCursor.<init>(ParquetHiveRecordCursor.java:157)
        at com.facebook.presto.hive.parquet.ParquetRecordCursorProvider.createRecordCursor(ParquetRecordCursorProvider.java:92)
        at com.facebook.presto.hive.HivePageSourceProvider.createHivePageSource(HivePageSourceProvider.java:157)
        at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:87)
        at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:44)
        at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:56)
        at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:234)
        at com.facebook.presto.operator.Driver.processInternal(Driver.java:416)
        at com.facebook.presto.operator.Driver.processFor(Driver.java:310)
        at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:622)
        at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:586)
        at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:734)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Filesystem closed
        at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2105)
        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
        at com.facebook.presto.hive.parquet.HdfsParquetDataSource.lambda$buildHdfsParquetDataSource$0(HdfsParquetDataSource.java:100)
        at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138)
        at com.facebook.presto.hive.parquet.HdfsParquetDataSource.buildHdfsParquetDataSource(HdfsParquetDataSource.java:99)
        ... 16 more
vage7 commented

Is there any update for this? Did you fix it or did you embed the above fix in any released version of presto code? I have the same problem when I run the first query of the day.

  • 👍1
rraghav13 commented

Have the same issue. The first query to Hive returns that error but the subsequent queries work ok. Any idea why? Hive 1.1.0-cdh5.9.1.

electrum commented

@arhimondr and I were chatting about this. UserGroupInformation has a checkTGTAndReloginFromKeytab() that we might be able to call to refresh the credentials and thus avoid the need to ever close a file system.

electrum commented

Fixed in #23

pi9nc commented

hi. after applying this patch, i still stuck into this issue and presto version is 0.183

any test result for this patch?

electrum commented

@pi9nc We should have a fix here: prestodb/presto#9079