prestodb/presto-hadoop-apache2

Presto Hive plugin is missing libraries needed for Hadoop KMS

Opened this issue · 3 comments

Deleted user commented

When Hadoop KMS is used, Presto Hive plugin calls org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL which fails with the current master.

Hadoop KMS calls use URIBuilder class which is missing in shaded version of hadoop libraries.

I have been able to fix this issue in https://github.com/vlkodev/presto-hadoop-apache2/commit/06b7b29060c40ec22318860696b6ab1677b5f84d

Steps I have done:

  1. Included httpclient library which contains URIBuilder class.
  2. After step 1 jackson dependencies were still missing.
  3. Included (removed exclusion) jackson-core-asl, jackson-mapper-asl

Would you consider pulling my change to include missing libraries?

URIBuilder class missing:

2016-06-20T12:13:24.190Z        ERROR   remote-task-callback-7  com.facebook.presto.execution.StageStateMachine Stage 20160620_121321_00003_nfw2i.1 failed
java.lang.NoClassDefFoundError: com/facebook/presto/hadoop/$internal/org/apache/http/client/utils/URIBuilder
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL(KMSClientProvider.java:433)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:773)
        at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
        at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1395)
        at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1465)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:305)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
        at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:108)
        at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
        at com.facebook.presto.hive.HiveUtil.lambda$createRecordReader$2(HiveUtil.java:166)
        at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:136)
        at com.facebook.presto.hive.HiveUtil.createRecordReader(HiveUtil.java:166)
        at com.facebook.presto.hive.GenericHiveRecordCursorProvider.createHiveRecordCursor(GenericHiveRecordCursorProvider.java:47)
        at com.facebook.presto.hive.HivePageSourceProvider.getHiveRecordCursor(HivePageSourceProvider.java:129)
        at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:107)
        at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:44)
        at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:48)
        at com.facebook.presto.operator.TableScanOperator.createSourceIfNecessary(TableScanOperator.java:268)
        at com.facebook.presto.operator.TableScanOperator.isFinished(TableScanOperator.java:210)
        at com.facebook.presto.operator.Driver.processInternal(Driver.java:375)
        at com.facebook.presto.operator.Driver.processFor(Driver.java:301)
        at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:618)
        at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:529)
        at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:665)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.facebook.presto.hadoop.$internal.org.apache.http.client.utils.URIBuilder
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at com.facebook.presto.server.PluginClassLoader.loadClass(PluginClassLoader.java:102)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

Jackson dependencies missing:

java.lang.NoClassDefFoundError: org/codehaus/jackson/map/ObjectMapper
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.writeJson(KMSClientProvider.java:216)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:513)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:779)
        at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
        at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1395)
        at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1465)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:305)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
        at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:108)
        at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
        at com.facebook.presto.hive.HiveUtil.lambda$createRecordReader$2(HiveUtil.java:166)
        at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:136)
        at com.facebook.presto.hive.HiveUtil.createRecordReader(HiveUtil.java:166)
        at com.facebook.presto.hive.GenericHiveRecordCursorProvider.createHiveRecordCursor(GenericHiveRecordCursorProvider.java:47)
        at com.facebook.presto.hive.HivePageSourceProvider.getHiveRecordCursor(HivePageSourceProvider.java:129)
        at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:107)
        at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:44)
        at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:48)
        at com.facebook.presto.operator.TableScanOperator.createSourceIfNecessary(TableScanOperator.java:268)
        at com.facebook.presto.operator.TableScanOperator.isFinished(TableScanOperator.java:210)
        at com.facebook.presto.operator.Driver.processInternal(Driver.java:375)
        at com.facebook.presto.operator.Driver.processFor(Driver.java:301)
        at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:618)
        at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:529)
        at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:665)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.map.ObjectMapper
        at com.facebook.presto.server.PluginClassLoader.loadClass(PluginClassLoader.java:106)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 32 more
electrum commented

We are fine with fixing this. Can you send a pull request?

Deleted user commented

Sure, pull request: #16

electrum commented

This will be resolved by prestodb/presto#6366