编译完整版后运行spark-xsql报异常
Closed this issue · 4 comments
环境信息：
CentOS Linux release 7.4.1708 (Core)
git branch
- （分离自 v0.6.1）
master
java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
echo $JAVA_HOME
/usr/java/jdk1.8.0_181-cloudera
配置信息：
vim ./conf/xsql.conf
spark.xsql.datasources default
spark.xsql.default.database test
spark.xsql.datasource.default.type mysql
spark.xsql.datasource.default.url jdbc:mysql://10.10.1.41:3306
spark.xsql.datasource.default.user test
spark.xsql.datasource.default.password test@123
spark.xsql.datasource.default.version 5.7.22
执行信息：
./spark-xsql
20/01/08 21:20:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/01/08 21:20:11 INFO SparkContext: Running Spark version 2.4.3
20/01/08 21:20:11 INFO SparkContext: Submitted application: org.apache.spark.sql.xsql.shell.SparkXSQLShell
20/01/08 21:20:11 INFO SecurityManager: Changing view acls to: root
20/01/08 21:20:11 INFO SecurityManager: Changing modify acls to: root
20/01/08 21:20:11 INFO SecurityManager: Changing view acls groups to:
20/01/08 21:20:11 INFO SecurityManager: Changing modify acls groups to:
20/01/08 21:20:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
20/01/08 21:20:11 INFO Utils: Successfully started service 'sparkDriver' on port 38039.
20/01/08 21:20:11 INFO SparkEnv: Registering MapOutputTracker
20/01/08 21:20:11 INFO SparkEnv: Registering BlockManagerMaster
20/01/08 21:20:11 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/01/08 21:20:11 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/01/08 21:20:11 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-799b3235-6aa8-4068-b013-f5caa4614c90
20/01/08 21:20:11 INFO MemoryStore: MemoryStore started with capacity 2.5 GB
20/01/08 21:20:11 INFO SparkEnv: Registering OutputCommitCoordinator
20/01/08 21:20:11 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/01/08 21:20:11 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://hdp-client-02:4040
20/01/08 21:20:11 INFO SparkContext: Added JAR file:/data/tools/xsql-0.6.0/xsql-0.6.1-bin-spark-2.4.3/jars/xsql-shell_2.11-0.6.1-SNAPSHOT.jar at spark://hdp-client-02:38039/jars/xsql-shell_2.11-0.6.1-SNAPSHOT.jar with timestamp 1578489611959
20/01/08 21:20:12 INFO Executor: Starting executor ID driver on host localhost
20/01/08 21:20:12 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45532.
20/01/08 21:20:12 INFO NettyBlockTransferService: Server created on hdp-client-02:45532
20/01/08 21:20:12 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/01/08 21:20:12 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, hdp-client-02, 45532, None)
20/01/08 21:20:12 INFO BlockManagerMasterEndpoint: Registering block manager hdp-client-02:45532 with 2.5 GB RAM, BlockManagerId(driver, hdp-client-02, 45532, None)
20/01/08 21:20:12 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, hdp-client-02, 45532, None)
20/01/08 21:20:12 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, hdp-client-02, 45532, None)
20/01/08 21:20:12 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/data/tools/xsql-0.6.0/xsql-0.6.1-bin-spark-2.4.3/bin/spark-warehouse').
20/01/08 21:20:12 INFO SharedState: Warehouse path is 'file:/data/tools/xsql-0.6.0/xsql-0.6.1-bin-spark-2.4.3/bin/spark-warehouse'.
20/01/08 21:20:13 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
20/01/08 21:20:13 INFO XSQLExternalCatalog: reading xsql configuration from /data/tools/xsql-0.6.0/xsql-0.6.1-bin-spark-2.4.3/conf/xsql.conf
20/01/08 21:20:13 INFO XSQLExternalCatalog: parse data source default
Exception in thread "main" java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.xsql.XSQLExternalCatalog':
at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:223)
at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:104)
at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:103)
at org.apache.spark.sql.xsql.XSQLSessionStateBuilder.externalCatalog(XSQLSessionStateBuilder.scala:60)
at org.apache.spark.sql.xsql.XSQLSessionStateBuilder.catalog$lzycompute(XSQLSessionStateBuilder.scala:73)
at org.apache.spark.sql.xsql.XSQLSessionStateBuilder.catalog(XSQLSessionStateBuilder.scala:71)
at org.apache.spark.sql.xsql.XSQLSessionStateBuilder.catalog(XSQLSessionStateBuilder.scala:57)
at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anonfun$build$1.apply(BaseSessionStateBuilder.scala:291)
at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anonfun$build$1.apply(BaseSessionStateBuilder.scala:291)
at org.apache.spark.sql.internal.SessionState.catalog$lzycompute(SessionState.scala:77)
at org.apache.spark.sql.internal.SessionState.catalog(SessionState.scala:77)
at org.apache.spark.sql.xsql.ResolveScanSingleTable.(XSQLStrategies.scala:69)
at org.apache.spark.sql.xsql.shell.SparkXSQLShell$.main(SparkXSQLShell.scala:55)
at org.apache.spark.sql.xsql.shell.SparkXSQLShell.main(SparkXSQLShell.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 305 milliseconds ago. The last packet sent successfully to the server was 296 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
at com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:201)
at com.mysql.jdbc.MysqlIO.negotiateSSLConnection(MysqlIO.java:4912)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1663)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1224)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2190)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2221)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2016)
at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:776)
at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.spark.sql.xsql.manager.MysqlManager.getConnect(MysqlManager.scala:75)
at org.apache.spark.sql.xsql.manager.MysqlManager.cacheDatabase(MysqlManager.scala:152)
at org.apache.spark.sql.xsql.DataSourceManager$class.parse(DataSourceManager.scala:202)
at org.apache.spark.sql.xsql.manager.MysqlManager.parse(MysqlManager.scala:51)
at org.apache.spark.sql.xsql.XSQLExternalCatalog.addDataSource(XSQLExternalCatalog.scala:439)
at org.apache.spark.sql.xsql.XSQLExternalCatalog$$anonfun$setupAndInitMetadata$4.apply(XSQLExternalCatalog.scala:172)
at org.apache.spark.sql.xsql.XSQLExternalCatalog$$anonfun$setupAndInitMetadata$4.apply(XSQLExternalCatalog.scala:162)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.sql.xsql.XSQLExternalCatalog.setupAndInitMetadata(XSQLExternalCatalog.scala:162)
at org.apache.spark.sql.xsql.XSQLExternalCatalog.(XSQLExternalCatalog.scala:119)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:214)
... 25 more
Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1964)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:328)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:186)
... 58 more
Caused by: java.security.cert.CertificateException: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at com.mysql.jdbc.ExportControlled$X509TrustManagerWrapper.checkServerTrusted(ExportControlled.java:302)
at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:992)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1596)
... 66 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:154)
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80)
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
at com.mysql.jdbc.ExportControlled$X509TrustManagerWrapper.checkServerTrusted(ExportControlled.java:295)
... 68 more
20/01/08 21:20:13 INFO SparkContext: Invoking stop() from shutdown hook
20/01/08 21:20:13 INFO SparkUI: Stopped Spark web UI at http://hdp-client-02:4040
20/01/08 21:20:13 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/01/08 21:20:13 INFO MemoryStore: MemoryStore cleared
20/01/08 21:20:13 INFO BlockManager: BlockManager stopped
20/01/08 21:20:13 INFO BlockManagerMaster: BlockManagerMaster stopped
20/01/08 21:20:14 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/01/08 21:20:14 INFO SparkContext: Successfully stopped SparkContext
20/01/08 21:20:14 INFO ShutdownHookManager: Shutdown hook called
20/01/08 21:20:14 INFO ShutdownHookManager: Deleting directory /tmp/spark-7f618cfa-d370-4a88-a6c5-8effc8f275e2
20/01/08 21:20:14 INFO ShutdownHookManager: Deleting directory /tmp/spark-a7925d7e-4b42-444d-bcc0-be1019ea63c7
Could you check if the MySQL server is OK ?
mysql server is ok, mysql-client can connect to this server
Please use useSSL=false
Great！thank you