[exchangis1.0.0]执行任务报错Run Error Message:java.lang.reflect.InvocationTargetException java.lang.reflect.InvocationTargetException: null
Culucy opened this issue · 2 comments
Culucy commented
[exchangis-server (1).log](https://github.com/WeBankFinTech/Exchangis/files/9897566/exchangis-server.1.log)
1. Listening for transport dt_socket at address: 37946
2. 2022-10-31 10:34:06.234 INFO [EngineConn-Asyn-Thread-Thread-0] org.apache.linkis.engineconn.executor.listener.EngineConnAsyncListenerBus 41 info - EngineConn-Asyn-Thread-Thread-0 begin.
3. 2022-10-31 10:34:06.386 INFO [EngineConn-Asyn-Thread-Thread-0] org.apache.linkis.engineconn.acessible.executor.service.DefaultManagerService 41 info - success to send engine heartbeat report to localhost:9101,status:Running,msg:null
4. 2022-10-31 10:34:06.623 INFO [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor 41 info - Try to execute params.{sqoop.args.password=Lucy867419., sqoop.args.driver=com.mysql.jdbc.Driver, sqoop.args.null.string=\\N, sqoop.args.hcatalog.partition.keys=ds, sqoop.args.columns=id,name,age,grade, sqoop.args.table=user, sqoop.args.where=1=1, sqoop.mode=import, sqoop.args.num.mappers=1, sqoop.args.hcatalog.table=user, sqoop.args.hcatalog.partition.values=1667097232847, sqoop.args.username=root, sqoop.args.fields.terminated.by=�, sqoop.args.null.non.string=\\N, sqoop.args.connect=jdbc:mysql://127.0.0.1:3306/test, sqoop.args.hcatalog.database=test_ind}
5. 2022-10-31 10:34:06.638 INFO [EngineConn-Asyn-Thread-Thread-0] org.apache.linkis.engineconn.acessible.executor.service.DefaultManagerService 41 info - success to send engine heartbeat report to localhost:9101,status:Busy,msg:null
6. 2022-10-31 10:34:06.655 INFO [EngineConn-Asyn-Thread-Thread-0] org.apache.linkis.engineconn.acessible.executor.service.DefaultManagerService 41 info - success to send engine heartbeat report to localhost:9101,status:Running,msg:null
7. 2022-10-31 10:34:06.770 WARN [Linkis-Default-Scheduler-Thread-1] org.apache.sqoop.tool.BaseSqoopTool 1021 applyCredentialsOptions - Setting your password on the command-line is insecure. Consider using -P instead.
8. 2022-10-31 10:34:06.772 WARN [Linkis-Default-Scheduler-Thread-1] org.apache.sqoop.tool.BaseSqoopTool 1530 validateHCatalogOptions - Output field/record delimiter options are not useful in HCatalog jobs for most of the output types except text based formats is text. It is better to use --hive-import in those cases. For non text formats,
9. 2022-10-31 10:34:06.818 WARN [Linkis-Default-Scheduler-Thread-1] org.apache.sqoop.ConnFactory 132 getManager - Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
10. 2022-10-31 10:34:06.824 INFO [Linkis-Default-Scheduler-Thread-1] org.apache.sqoop.manager.SqlManager 98 initOptionDefaults - Using default fetchSize of 1000
11. 2022-10-31 10:34:06.824 INFO [Linkis-Default-Scheduler-Thread-1] org.apache.sqoop.tool.CodeGenTool 92 generateORM - Beginning code generation
12. 2022-10-31 10:34:06.959 INFO [PollingServerListUpdater-0] com.netflix.config.ChainedDynamicProperty 115 checkAndFlip - Flipping property: linkis-cg-engineconnmanager.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
13. 2022-10-31 10:34:07.343 INFO [PollingServerListUpdater-1] com.netflix.config.ChainedDynamicProperty 115 checkAndFlip - Flipping property: linkis-cg-linkismanager.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
14. 2022-10-31 10:34:07.356 INFO [Linkis-Default-Scheduler-Thread-1] org.apache.sqoop.manager.SqlManager 757 execute - Executing SQL statement: SELECT t.* FROM user AS t WHERE 1=0
15. 2022-10-31 10:34:07.365 INFO [Linkis-Default-Scheduler-Thread-1] org.apache.sqoop.orm.CompilationManager 85 findHadoopJars - $HADOOP_MAPRED_HOME is not set
16. 2022-10-31 10:34:08.768 INFO [Linkis-Default-Scheduler-Thread-1] org.apache.sqoop.orm.CompilationManager 330 jar - Writing jar file: /tmp/sqoop-hadoop/compile/60075ad13be812f20536b73e1ab8070e/user.jar
17. org.apache.linkis.engineconnplugin.sqoop.client.utils.JarLoader@1ffd1daa
18. 2022-10-31 10:34:08.784 INFO [Linkis-Default-Scheduler-Thread-1] org.apache.sqoop.mapreduce.ImportJobBase 235 runImport - Beginning import of user
19. 2022-10-31 10:34:08.946 WARN [Linkis-Default-Scheduler-Thread-1] org.apache.hadoop.util.NativeCodeLoader 62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20. 2022-10-31 10:34:08.956 INFO [Linkis-Default-Scheduler-Thread-1] org.apache.hadoop.conf.Configuration.deprecation 1173 warnOnceIfDeprecated - mapred.jar is deprecated. Instead, use mapreduce.job.jar
21. 2022-10-31 10:34:08.972 ERROR [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconnplugin.sqoop.client.LinkisSqoopClient 69 run - Run Error Message:java.lang.reflect.InvocationTargetException java.lang.reflect.InvocationTargetException: null
22. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_342]
23. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_342]
24. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_342]
25. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_342]
26. at org.apache.linkis.engineconnplugin.sqoop.client.LinkisSqoopClient.run(LinkisSqoopClient.java:67) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
27. at org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor$$anonfun$runSqoop$1.apply$mcI$sp(SqoopOnceCodeExecutor.scala:75) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
28. at org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor$$anonfun$runSqoop$1.apply(SqoopOnceCodeExecutor.scala:71) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
29. at org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor$$anonfun$runSqoop$1.apply(SqoopOnceCodeExecutor.scala:71) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
30. at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40) ~[linkis-common-1.1.1.jar:1.1.1]
31. at org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor.runSqoop(SqoopOnceCodeExecutor.scala:76) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
32. at org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor$$anon$1.run(SqoopOnceCodeExecutor.scala:56) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
33. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_342]
34. at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_342]
35. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_342]
36. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_342]
37. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342]
38. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342]
39. at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342]
40. Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatOutputFormat
41. at org.apache.sqoop.mapreduce.DataDrivenImportJob.getOutputFormatClass(DataDrivenImportJob.java:178) ~[sqoop-1.4.6-hadoop200.jar:1.1.1]
42. at org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBase.java:94) ~[sqoop-1.4.6-hadoop200.jar:1.1.1]
43. at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:259) ~[sqoop-1.4.6-hadoop200.jar:1.1.1]
44. at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673) ~[sqoop-1.4.6-hadoop200.jar:?]
45. at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497) ~[sqoop-1.4.6-hadoop200.jar:?]
46. at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) ~[sqoop-1.4.6-hadoop200.jar:?]
47. at org.apache.linkis.engineconnplugin.sqoop.client.Sqoop.run(Sqoop.java:157) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
48. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) ~[hadoop-common-2.7.2.jar:?]
49. at org.apache.linkis.engineconnplugin.sqoop.client.Sqoop.runSqoop(Sqoop.java:190) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
50. at org.apache.linkis.engineconnplugin.sqoop.client.Sqoop.runTool(Sqoop.java:237) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
142. at org.apache.linkis.engineconnplugin.sqoop.client.Sqoop.runTool(Sqoop.java:297) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
143. at org.apache.linkis.engineconnplugin.sqoop.client.Sqoop.main(Sqoop.java:301) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
144. ... 18 more
145. Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.mapreduce.HCatOutputFormat
146. at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_342]
147. at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_342]
148. at org.apache.linkis.engineconnplugin.sqoop.client.utils.JarLoader.loadClass(JarLoader.java:158) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
149. at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_342]
150. at org.apache.sqoop.mapreduce.DataDrivenImportJob.getOutputFormatClass(DataDrivenImportJob.java:178) ~[sqoop-1.4.6-hadoop200.jar:1.1.1]
151. at org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBase.java:94) ~[sqoop-1.4.6-hadoop200.jar:1.1.1]
152. at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:259) ~[sqoop-1.4.6-hadoop200.jar:1.1.1]
153. at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673) ~[sqoop-1.4.6-hadoop200.jar:?]
154. at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497) ~[sqoop-1.4.6-hadoop200.jar:?]
155. at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) ~[sqoop-1.4.6-hadoop200.jar:?]
156. at org.apache.linkis.engineconnplugin.sqoop.client.Sqoop.run(Sqoop.java:157) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
157. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) ~[hadoop-common-2.7.2.jar:?]
158. at org.apache.linkis.engineconnplugin.sqoop.client.Sqoop.runSqoop(Sqoop.java:190) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
159. at org.apache.linkis.engineconnplugin.sqoop.client.Sqoop.runTool(Sqoop.java:237) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
160. at org.apache.linkis.engineconnplugin.sqoop.client.Sqoop.runTool(Sqoop.java:297) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
161. at org.apache.linkis.engineconnplugin.sqoop.client.Sqoop.main(Sqoop.java:301) ~[linkis-engineplugin-sqoop-1.1.1.jar:1.1.1]
162. ... 18 more
163.
164. 2022-10-31 10:34:08.985 ERROR [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor 62 error - SqoopOnceApp_0 has failed with old status Running, now stop it.
165. 2022-10-31 10:34:08.987 INFO [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor 53 transition - Waitiing lock release, to change status Running=>Failed.
166. 2022-10-31 10:34:08.988 INFO [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor 55 transition - Finished wait lock release, to change status Running=>Failed.
167. 2022-10-31 10:34:08.988 INFO [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor 41 info - org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor@4aee166d changed status Running => Failed.
168. 2022-10-31 10:34:08.990 ERROR [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconn.once.executor.execution.OnceEngineConnExecution 58 error - Unknown reason.
169. 2022-10-31 10:34:08.991 ERROR [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor 62 error - SqoopOnceApp_0 has failed with old status Failed, now stop it.
170. 2022-10-31 10:34:08.997 WARN [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor 50 warn - Executor(SqoopOnceApp_0) exit by close.
171. 2022-10-31 10:34:08.997 WARN [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor 50 warn - Executor(SqoopOnceApp_0) exit by close.
172. 2022-10-31 10:34:08.998 INFO [Linkis-Default-Scheduler-Thread-1] org.apache.linkis.engineconnplugin.sqoop.executor.SqoopOnceCodeExecutor 41 info - All codes completed, now to stop SqoopEngineConn.
173. 2022-10-31 10:34:08.999 INFO [Thread-31] com.netflix.loadbalancer.PollingServerListUpdater 53 run - Shutting down the Executor Pool for PollingServerListUpdater
174. 2022-10-31 10:34:09.000 INFO [EngineConn-Asyn-Thread-Thread-1] org.apache.linkis.engineconn.executor.listener.EngineConnAsyncListenerBus 41 info - EngineConn-Asyn-Thread-Thread-1 begin.
175. 2022-10-31 10:34:09.014 INFO [EngineConn-Asyn-Thread-Thread-0] org.apache.linkis.engineconn.acessible.executor.service.DefaultManagerService 41 info - success to send engine heartbeat report to localhost:9101,status:Failed,msg:null
176. 2022-10-31 10:34:09.018 INFO [EngineConn-Asyn-Thread-Thread-1] org.apache.linkis.engineconn.acessible.executor.service.DefaultManagerService 41 info - success to send engine heartbeat report to localhost:9101,status:Failed,msg:null
jefftlin commented
I found "Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatOutputFormat 41. at org.apache.sqoop.mapreduce.DataDrivenImportJob.getOutputFormatClass(DataDrivenImportJob.java:178)" and "Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.mapreduce.HCatOutputFormat" in your log. Have you installed hive environment correctly?
jefftlin commented
The latest version of exchangis is 1.1.2, the latest dss version is 1.1.2, and the latest linkis version is 1.4.0