melin/spark-jobserver

启动 jobserver 失败

Opened this issue · 3 comments

yinlice commented

java.lang.RuntimeException: 启动 jobserver 失败, Spark App Status：FAILED, 失败原因: java.lang.RuntimeException: Exception in thread "main" java.lang.IllegalArgumentException: Attempt to add (file:///root/tmp/spark-jobserver-config/Test/hive-site.xml) multiple times to the distributed cache.
at org.apache.spark.launcher.OutputRedirector.redirect(OutputRedirector.java:67)
at java.lang.Thread.run(Thread.java:750)

    at io.github.melin.spark.jobserver.deployment.YarnSparkDriverDeployer.buildJobServer(YarnSparkDriverDeployer.java:134) ~[conf/:?]
    at io.github.melin.spark.jobserver.monitor.DriverPoolManager.lambda$startMinJobServer$1(DriverPoolManager.java:131) ~[conf/:?]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_362]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_362]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) ~[hadoop-common-3.3.4.jar:?]
    at io.github.melin.spark.jobserver.support.ClusterManager.runSecured(ClusterManager.java:142) ~[conf/:?]
    at io.github.melin.spark.jobserver.monitor.DriverPoolManager.startMinJobServer(DriverPoolManager.java:130) ~[conf/:?]
    at io.github.melin.spark.jobserver.monitor.DriverPoolManager.lambda$afterPropertiesSet$0(DriverPoolManager.java:72) ~[conf/:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_362]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_362]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_362]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_362]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
    at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
yinlice commented

这个是因为有配置没弄对吗

melin commented

查看一下yarn log:
yarn logs -applicationId xxxx

yinlice commented

查看一下yarn log: yarn logs -applicationId xxxx

没有提交上去.