error to build spark error
Arnold1 opened this issue · 13 comments
Hi,
i tried to build spark as follows:
git clone https://g...content-available-to-author-only...b.com/apache-spark-on-k8s/spark
cd spark
build/mvn compile -Pkubernetes -pl resource-managers/kubernetes/core -am -DskipTests -X
and get this error:
[INFO] Spark Project Parent POM ........................... SUCCESS [ 1.926 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 2.234 s]
[INFO] Spark Project Networking ........................... FAILURE [ 0.056 s]
[INFO] Spark Project Shuffle Streaming Service ............ SKIPPED
[INFO] Spark Project Unsafe ............................... SKIPPED
[INFO] Spark Project Launcher ............................. SKIPPED
[INFO] Spark Project Core ................................. SKIPPED
[INFO] Spark Project Kubernetes ........................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.737 s
[INFO] Finished at: 2017-12-29T20:31:28-05:00
[INFO] Final Memory: 38M/335M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project spark-network-common_2.11: Could not resolve dependencies for project org.apache.spark:spark-network-common_2.11:jar:2.2.0-k8s-0.5.0: Failure to find org.apache.spark:spark-tags_2.11:jar:tests:2.2.0-k8s-0.5.0 in https://r...content-available-to-author-only...n.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project spark-network-common_2.11: Could not resolve dependencies for project org.apache.spark:spark-network-common_2.11:jar:2.2.0-k8s-0.5.0: Failure to find org.apache.spark:spark-tags_2.11:jar:tests:2.2.0-k8s-0.5.0 in https://r...content-available-to-author-only...n.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:221)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:246)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:200)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project org.apache.spark:spark-network-common_2.11:jar:2.2.0-k8s-0.5.0: Failure to find org.apache.spark:spark-tags_2.11:jar:tests:2.2.0-k8s-0.5.0 in https://r...content-available-to-author-only...n.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:208)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:195)
... 23 more
Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Failure to find org.apache.spark:spark-tags_2.11:jar:tests:2.2.0-k8s-0.5.0 in https://r...content-available-to-author-only...n.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:393)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:202)
... 24 more
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Failure to find org.apache.spark:spark-tags_2.11:jar:tests:2.2.0-k8s-0.5.0 in https://r...content-available-to-author-only...n.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:453)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:255)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:376)
... 25 more
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Failure to find org.apache.spark:spark-tags_2.11:jar:tests:2.2.0-k8s-0.5.0 in https://r...content-available-to-author-only...n.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
at org.eclipse.aether.internal.impl.DefaultUpdateCheckManager.newException(DefaultUpdateCheckManager.java:240)
at org.eclipse.aether.internal.impl.DefaultUpdateCheckManager.checkArtifact(DefaultUpdateCheckManager.java:215)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.gatherDownloads(DefaultArtifactResolver.java:594)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:512)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:430)
... 27 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://c...content-available-to-author-only...e.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :spark-network-common_2.11
how to fix it?
You need to install dependencies first. Run mvn install
@ifilonenko i did that, but got this issue:
$ man install
...
...
Run completed in 23 minutes, 35 seconds.
Total number of tests run: 3160
Suites: completed 189, aborted 0
Tests: succeeded 3156, failed 4, canceled 0, ignored 53, pending 0
*** 4 TESTS FAILED ***
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 21.614 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 9.143 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 23.532 s]
[INFO] Spark Project Networking ........................... SUCCESS [01:00 min]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 15.098 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 18.472 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 23.050 s]
[INFO] Spark Project Core ................................. SUCCESS [21:34 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 40.734 s]
[INFO] Spark Project GraphX ............................... SUCCESS [01:37 min]
[INFO] Spark Project Streaming ............................ SUCCESS [06:07 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [14:44 min]
[INFO] Spark Project SQL .................................. FAILURE [30:00 min]
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External Flume Assembly .............. SKIPPED
[INFO] Spark Integration for Kafka 0.8 .................... SKIPPED
[INFO] Kafka 0.10 Source for Structured Streaming ......... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Project External Kafka Assembly .............. SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:17 h
[INFO] Finished at: 2017-12-30T18:55:18-05:00
[INFO] Final Memory: 90M/1444M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0:test (test) on project spark-sql_2.11: There are test failures -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :spark-sql_2.11
@ifilonenko you mean the following instead?
build/mvn install -Pkubernetes -pl resource-managers/kubernetes/core -am -DskipTests
build/mvn compile -T 4C -Pkubernetes -pl resource-managers/kubernetes/core -am -DskipTests
than i can do?
dev/make-distribution.sh --tgz -Phadoop-2.7 -Pkubernetes
Yeah
@ifilonenko ok i did that 3 commands.
now i run the following. the spark output from the example still give me failed...any idea?
i start minikube:
minikube start
get minikube ip:
$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
run.sh script
# $ kubectl cluster-info
k8s_apiserver_host="192.168.99.100"
k8s_apiserver_port="8443"
cwd=`pwd`
echo $k8s_apiserver_host
echo $k8s_apiserver_port
echo $cwd
spark/bin/spark-submit \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--master k8s://https://$k8s_apiserver_host:$k8s_apiserver_port \
--kubernetes-namespace default \
--conf spark.executor.instances=5 \
--conf spark.app.name=spark-pi \
--conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 \
local:///opt/spark/examples/target/scala-2.11/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar
here i call the run.sh script
$ ./run.sh
192.168.99.100
8443
/Users/arnold/run_spark_on_k8s
2017-12-31 12:12:26 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1514740344980-driver
namespace: default
labels: spark-app-selector -> spark-7492d2e12cd5417fb454e527e976876a, spark-role -> driver
pod uid: c5850d2e-ee4d-11e7-9c13-08002718fe97
creation time: 2017-12-31T17:12:25Z
service account name: default
volumes: spark-local-dir-0-spark-c5c62c8d-7772-40af-81bb-3ffff61a8fba, default-token-nxcnb
node name: N/A
start time: N/A
container images: N/A
phase: Pending
status: []
2017-12-31 12:12:26 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1514740344980-driver
namespace: default
labels: spark-app-selector -> spark-7492d2e12cd5417fb454e527e976876a, spark-role -> driver
pod uid: c5850d2e-ee4d-11e7-9c13-08002718fe97
creation time: 2017-12-31T17:12:25Z
service account name: default
volumes: spark-local-dir-0-spark-c5c62c8d-7772-40af-81bb-3ffff61a8fba, default-token-nxcnb
node name: minikube
start time: N/A
container images: N/A
phase: Pending
status: []
2017-12-31 12:12:26 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1514740344980-driver
namespace: default
labels: spark-app-selector -> spark-7492d2e12cd5417fb454e527e976876a, spark-role -> driver
pod uid: c5850d2e-ee4d-11e7-9c13-08002718fe97
creation time: 2017-12-31T17:12:25Z
service account name: default
volumes: spark-local-dir-0-spark-c5c62c8d-7772-40af-81bb-3ffff61a8fba, default-token-nxcnb
node name: minikube
start time: 2017-12-31T17:12:25Z
container images: kubespark/spark-driver:v2.2.0-kubernetes-0.5.0
phase: Pending
status: [ContainerStatus(containerID=null, image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={}), additionalProperties={}), additionalProperties={})]
2017-12-31 12:12:26 INFO Client:54 - Waiting for application spark-pi to finish...
2017-12-31 12:12:26 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1514740344980-driver
namespace: default
labels: spark-app-selector -> spark-7492d2e12cd5417fb454e527e976876a, spark-role -> driver
pod uid: c5850d2e-ee4d-11e7-9c13-08002718fe97
creation time: 2017-12-31T17:12:25Z
service account name: default
volumes: spark-local-dir-0-spark-c5c62c8d-7772-40af-81bb-3ffff61a8fba, default-token-nxcnb
node name: minikube
start time: 2017-12-31T17:12:25Z
container images: kubespark/spark-driver:v2.2.0-kubernetes-0.5.0
phase: Running
status: [ContainerStatus(containerID=docker://d80aa779d02d0964c1e61eeb167bae1e75c9da77720b2940d65ef555eaee4839, image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0, imageID=docker-pullable://kubespark/spark-driver@sha256:7539c28c4dbfc55addf5c3f439d26b964c4b04cd780a066c708242867356eb0b, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=true, restartCount=0, state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2017-12-31T17:12:25Z, additionalProperties={}), additionalProperties={}), terminated=null, waiting=null, additionalProperties={}), additionalProperties={})]
2017-12-31 12:12:27 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1514740344980-driver
namespace: default
labels: spark-app-selector -> spark-7492d2e12cd5417fb454e527e976876a, spark-role -> driver
pod uid: c5850d2e-ee4d-11e7-9c13-08002718fe97
creation time: 2017-12-31T17:12:25Z
service account name: default
volumes: spark-local-dir-0-spark-c5c62c8d-7772-40af-81bb-3ffff61a8fba, default-token-nxcnb
node name: minikube
start time: 2017-12-31T17:12:25Z
container images: kubespark/spark-driver:v2.2.0-kubernetes-0.5.0
phase: Failed
status: [ContainerStatus(containerID=docker://d80aa779d02d0964c1e61eeb167bae1e75c9da77720b2940d65ef555eaee4839, image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0, imageID=docker-pullable://kubespark/spark-driver@sha256:7539c28c4dbfc55addf5c3f439d26b964c4b04cd780a066c708242867356eb0b, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://d80aa779d02d0964c1e61eeb167bae1e75c9da77720b2940d65ef555eaee4839, exitCode=1, finishedAt=Time(time=2017-12-31T17:12:26Z, additionalProperties={}), message=null, reason=Error, signal=null, startedAt=Time(time=2017-12-31T17:12:25Z, additionalProperties={}), additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})]
2017-12-31 12:12:27 INFO LoggingPodStatusWatcherImpl:54 - Container final statuses:
Container name: spark-kubernetes-driver
Container image: kubespark/spark-driver:v2.2.0-kubernetes-0.5.0
Container state: Terminated
Exit code: 1
2017-12-31 12:12:27 INFO Client:54 - Application spark-pi finished.
We need pod logs, it failed for some reason. kubectl get logs <pod_name>
or look on dashboard after running kubectl proxy
see that worked now, all good?
2017-12-31 18:43:53 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1514763831725-driver
namespace: default
labels: spark-app-selector -> spark-03ad4433db5d46a8809d79bae90f6b1d, spark-role -> driver
pod uid: 75aa8958-ee84-11e7-85ee-080027117b36
creation time: 2017-12-31T23:43:53Z
service account name: default
volumes: spark-local-dir-0-spark-51248d16-5032-417d-8e1e-b2ed5122e9c4, default-token-thhcr
node name: N/A
start time: N/A
container images: N/A
phase: Pending
status: []
2017-12-31 18:43:53 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1514763831725-driver
namespace: default
labels: spark-app-selector -> spark-03ad4433db5d46a8809d79bae90f6b1d, spark-role -> driver
pod uid: 75aa8958-ee84-11e7-85ee-080027117b36
creation time: 2017-12-31T23:43:53Z
service account name: default
volumes: spark-local-dir-0-spark-51248d16-5032-417d-8e1e-b2ed5122e9c4, default-token-thhcr
node name: minikube
start time: N/A
container images: N/A
phase: Pending
status: []
2017-12-31 18:43:53 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1514763831725-driver
namespace: default
labels: spark-app-selector -> spark-03ad4433db5d46a8809d79bae90f6b1d, spark-role -> driver
pod uid: 75aa8958-ee84-11e7-85ee-080027117b36
creation time: 2017-12-31T23:43:53Z
service account name: default
volumes: spark-local-dir-0-spark-51248d16-5032-417d-8e1e-b2ed5122e9c4, default-token-thhcr
node name: minikube
start time: 2017-12-31T23:43:53Z
container images: kubespark/spark-driver:v2.2.0-kubernetes-0.5.0
phase: Pending
status: [ContainerStatus(containerID=null, image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={}), additionalProperties={}), additionalProperties={})]
2017-12-31 18:43:53 INFO Client:54 - Waiting for application spark-pi to finish...
2017-12-31 18:46:51 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1514763831725-driver
namespace: default
labels: spark-app-selector -> spark-03ad4433db5d46a8809d79bae90f6b1d, spark-role -> driver
pod uid: 75aa8958-ee84-11e7-85ee-080027117b36
creation time: 2017-12-31T23:43:53Z
service account name: default
volumes: spark-local-dir-0-spark-51248d16-5032-417d-8e1e-b2ed5122e9c4, default-token-thhcr
node name: minikube
start time: 2017-12-31T23:43:53Z
container images: kubespark/spark-driver:v2.2.0-kubernetes-0.5.0
phase: Running
status: [ContainerStatus(containerID=docker://3c58f33aa48a083a63b4b93bde4f9814d0b9794ad9073d6b27b4d0266f770a1d, image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0, imageID=docker-pullable://kubespark/spark-driver@sha256:7539c28c4dbfc55addf5c3f439d26b964c4b04cd780a066c708242867356eb0b, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=true, restartCount=0, state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2017-12-31T23:46:51Z, additionalProperties={}), additionalProperties={}), terminated=null, waiting=null, additionalProperties={}), additionalProperties={})]
2017-12-31 18:47:25 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-1514763831725-driver
namespace: default
labels: spark-app-selector -> spark-03ad4433db5d46a8809d79bae90f6b1d, spark-role -> driver
pod uid: 75aa8958-ee84-11e7-85ee-080027117b36
creation time: 2017-12-31T23:43:53Z
service account name: default
volumes: spark-local-dir-0-spark-51248d16-5032-417d-8e1e-b2ed5122e9c4, default-token-thhcr
node name: minikube
start time: 2017-12-31T23:43:53Z
container images: kubespark/spark-driver:v2.2.0-kubernetes-0.5.0
phase: Succeeded
status: [ContainerStatus(containerID=docker://3c58f33aa48a083a63b4b93bde4f9814d0b9794ad9073d6b27b4d0266f770a1d, image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0, imageID=docker-pullable://kubespark/spark-driver@sha256:7539c28c4dbfc55addf5c3f439d26b964c4b04cd780a066c708242867356eb0b, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://3c58f33aa48a083a63b4b93bde4f9814d0b9794ad9073d6b27b4d0266f770a1d, exitCode=0, finishedAt=Time(time=2017-12-31T23:47:25Z, additionalProperties={}), message=null, reason=Completed, signal=null, startedAt=Time(time=2017-12-31T23:46:51Z, additionalProperties={}), additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})]
2017-12-31 18:47:25 INFO LoggingPodStatusWatcherImpl:54 - Container final statuses:
Container name: spark-kubernetes-driver
Container image: kubespark/spark-driver:v2.2.0-kubernetes-0.5.0
Container state: Terminated
Exit code: 0
2017-12-31 18:47:25 INFO Client:54 - Application spark-pi finished.
Check the logs:
$ kubectl logs `kubectl get pods --show-all | grep spark-pi | sort | head -n1 | awk '{print $1}'`
...
...
Pi is roughly 3.141315706578533
...
Looks good. Can I close this now?
last question. why do you need 5 instances? it seems it run out of cpus and can only run 3 instances...
--conf spark.executor.instances=5
This config is setting how many executors you want. It’s customizable. You do not have to set 5.
ok cool. closing it now.