Different Spark Versions

[ronniemo]

After cloning and bringing it up, the version of the spark cluster is 3.0.1 and not the same as the spark version 2.4.5 and the spark_test job fails.

Its running after adding spark 3.0.1.

The Log for the failed spark_test under Spark 2.4.5 here:

Reading local file: /usr/local/airflow/logs/spark-test/spark_job/2020-12-02T15:48:06.551084+00:00/2.log
[2020-12-02 15:50:15,406] {{taskinstance.py:655}} INFO - Dependencies all met for <TaskInstance: spark-test.spark_job 2020-12-02T15:48:06.551084+00:00 [queued]>
[2020-12-02 15:50:15,416] {{taskinstance.py:655}} INFO - Dependencies all met for <TaskInstance: spark-test.spark_job 2020-12-02T15:48:06.551084+00:00 [queued]>
[2020-12-02 15:50:15,416] {{taskinstance.py:866}} INFO -

[2020-12-02 15:50:15,416] {{taskinstance.py:867}} INFO - Starting attempt 2 of 2
[2020-12-02 15:50:15,416] {{taskinstance.py:868}} INFO -

[2020-12-02 15:50:15,426] {{taskinstance.py:887}} INFO - Executing <Task(SparkSubmitOperator): spark_job> on 2020-12-02T15:48:06.551084+00:00
[2020-12-02 15:50:15,427] {{standard_task_runner.py:52}} INFO - Started process 2010 to run task
[2020-12-02 15:50:15,448] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,448] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags/spark-test.py
[2020-12-02 15:50:15,460] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: spark-test.spark_job 2020-12-02T15:48:06.551084+00:00 [running]> 0f1bfb481b22
[2020-12-02 15:50:15,498] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,497] {{base_hook.py:84}} INFO - Using connection to: id: spark_default. Host: spark://spark, Port: 7077, Schema: None, Login: None, Password: None, extra: XXXXXXXX
[2020-12-02 15:50:15,499] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,498] {{spark_submit_hook.py:305}} INFO - Spark-Submit cmd: ['spark-submit', '--master', 'spark://spark:7077', '--conf', 'spark.master=spark://spark:7077', '--name', 'Spark Hello World', '--verbose', '--queue', 'root.default', '/usr/local/spark/app/hello-world.py', '/usr/local/spark/resources/data/airflow.cfg']
[2020-12-02 15:50:15,843] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,842] {{spark_submit_hook.py:436}} INFO - Using properties file: null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,877] {{spark_submit_hook.py:436}} INFO - Parsed arguments:
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - master spark://spark:7077
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - deployMode null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - executorMemory null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - executorCores null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - totalExecutorCores null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - propertiesFile null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - driverMemory null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - driverCores null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - driverExtraClassPath null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - driverExtraLibraryPath null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - driverExtraJavaOptions null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - supervise false
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - queue root.default
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - numExecutors null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - files null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - pyFiles null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - archives null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - mainClass null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - primaryResource file:/usr/local/spark/app/hello-world.py
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - name Spark Hello World
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - childArgs [/usr/local/spark/resources/data/airflow.cfg]
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - jars null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - packages null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - packagesExclusions null
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO - repositories null
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO - verbose true
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO - Spark properties used, including those specified through
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO - --conf and those from the properties file null:
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO - (spark.master,spark://spark:7077)
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:50:16,094] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,094] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[2020-12-02 15:50:16,458] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,458] {{spark_submit_hook.py:436}} INFO - Main class:
[2020-12-02 15:50:16,458] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,458] {{spark_submit_hook.py:436}} INFO - org.apache.spark.deploy.PythonRunner
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,458] {{spark_submit_hook.py:436}} INFO - Arguments:
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - file:/usr/local/spark/app/hello-world.py
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - null
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - /usr/local/spark/resources/data/airflow.cfg
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - Spark config:
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - (spark.master,spark://spark:7077)
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - (spark.app.name,Spark Hello World)
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - (spark.submit.deployMode,client)
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - Classpath elements:
[2020-12-02 15:50:16,460] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:50:16,460] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,460] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:50:16,460] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,460] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:50:16,697] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,697] {{spark_submit_hook.py:436}} INFO - Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
[2020-12-02 15:50:16,698] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,698] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SparkContext: Running Spark version 2.4.5
[2020-12-02 15:50:16,709] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,709] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SparkContext: Submitted application: Spark Hello World
[2020-12-02 15:50:16,733] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,733] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SecurityManager: Changing view acls to: airflow
[2020-12-02 15:50:16,734] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,733] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SecurityManager: Changing modify acls to: airflow
[2020-12-02 15:50:16,734] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,734] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SecurityManager: Changing view acls groups to:
[2020-12-02 15:50:16,734] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,734] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SecurityManager: Changing modify acls groups to:
[2020-12-02 15:50:16,734] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,734] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(airflow); groups with view permissions: Set(); users with modify permissions: Set(airflow); groups with modify permissions: Set()
[2020-12-02 15:50:16,899] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,899] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO Utils: Successfully started service 'sparkDriver' on port 35829.
[2020-12-02 15:50:16,911] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,911] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SparkEnv: Registering MapOutputTracker
[2020-12-02 15:50:16,919] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,919] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SparkEnv: Registering BlockManagerMaster
[2020-12-02 15:50:16,921] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,921] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
[2020-12-02 15:50:16,921] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,921] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
[2020-12-02 15:50:16,925] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,925] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-6d01f882-8c44-46db-a2ba-fc7c78b245fb
[2020-12-02 15:50:16,933] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,933] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
[2020-12-02 15:50:16,939] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,939] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SparkEnv: Registering OutputCommitCoordinator
[2020-12-02 15:50:17,022] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:17,022] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:17 INFO Utils: Successfully started service 'SparkUI' on port 4040.
[2020-12-02 15:50:17,046] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:17,046] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:17 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://0f1bfb481b22:4040
[2020-12-02 15:50:17,093] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:17,093] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:17 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://spark:7077...
[2020-12-02 15:50:17,120] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:17,120] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:17 INFO TransportClientFactory: Successfully created connection to spark/172.26.0.4:7077 after 16 ms (0 ms spent in bootstraps)
[2020-12-02 15:50:37,094] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:37,094] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:37 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://spark:7077...
[2020-12-02 15:50:57,094] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:57,094] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:57 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://spark:7077...
[2020-12-02 15:51:17,095] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,094] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
[2020-12-02 15:51:17,095] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,095] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 WARN StandaloneSchedulerBackend: Application ID is not initialized yet.
[2020-12-02 15:51:17,099] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,099] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41935.
[2020-12-02 15:51:17,099] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,099] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO NettyBlockTransferService: Server created on 0f1bfb481b22:41935
[2020-12-02 15:51:17,100] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,100] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO SparkUI: Stopped Spark web UI at http://0f1bfb481b22:4040
[2020-12-02 15:51:17,100] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,100] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
[2020-12-02 15:51:17,102] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,102] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO StandaloneSchedulerBackend: Shutting down all executors
[2020-12-02 15:51:17,104] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,104] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
[2020-12-02 15:51:17,106] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,106] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 WARN StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
[2020-12-02 15:51:17,108] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,108] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
[2020-12-02 15:51:17,115] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,115] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 0f1bfb481b22, 41935, None)
[2020-12-02 15:51:17,117] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,117] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManagerMasterEndpoint: Registering block manager 0f1bfb481b22:41935 with 366.3 MB RAM, BlockManagerId(driver, 0f1bfb481b22, 41935, None)
[2020-12-02 15:51:17,119] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,119] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 0f1bfb481b22, 41935, None)
[2020-12-02 15:51:17,119] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,119] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 0f1bfb481b22, 41935, None)
[2020-12-02 15:51:17,125] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,124] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO MemoryStore: MemoryStore cleared
[2020-12-02 15:51:17,125] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,125] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManager: BlockManager stopped
[2020-12-02 15:51:17,128] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,128] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManagerMaster: BlockManagerMaster stopped
[2020-12-02 15:51:17,129] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,129] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
[2020-12-02 15:51:17,144] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,144] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO SparkContext: Successfully stopped SparkContext
[2020-12-02 15:51:17,191] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,191] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 ERROR SparkContext: Error initializing SparkContext.
[2020-12-02 15:51:17,191] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,191] {{spark_submit_hook.py:436}} INFO - java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
[2020-12-02 15:51:17,191] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,191] {{spark_submit_hook.py:436}} INFO - at scala.Predef$.require(Predef.scala:224)
[2020-12-02 15:51:17,191] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,191] {{spark_submit_hook.py:436}} INFO - at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
[2020-12-02 15:51:17,191] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,191] {{spark_submit_hook.py:436}} INFO - at org.apache.spark.SparkContext.(SparkContext.scala:516)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at py4j.Gateway.invoke(Gateway.java:238)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at py4j.GatewayConnection.run(GatewayConnection.java:238)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at java.lang.Thread.run(Thread.java:748)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO SparkContext: SparkContext already stopped.
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - Traceback (most recent call last):
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - File "/usr/local/spark/app/hello-world.py", line 14, in
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - sc = SparkContext()
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - File "/usr/local/lib/python3.6/site-packages/pyspark/context.py", line 136, in init
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - conf, jsc, profiler_cls)
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - File "/usr/local/lib/python3.6/site-packages/pyspark/context.py", line 198, in _do_init
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - self._jsc = jsc or self._initialize_context(self._conf._jconf)
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - File "/usr/local/lib/python3.6/site-packages/pyspark/context.py", line 306, in _initialize_context
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - return self._jvm.JavaSparkContext(jconf)
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - File "/usr/local/lib/python3.6/site-packages/py4j/java_gateway.py", line 1525, in call
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - answer, self._gateway_client, None, self._fqn)
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - File "/usr/local/lib/python3.6/site-packages/py4j/protocol.py", line 328, in get_return_value
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - format(target_id, ".", name), value)
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - : java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at scala.Predef$.require(Predef.scala:224)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at org.apache.spark.SparkContext.(SparkContext.scala:516)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at py4j.Gateway.invoke(Gateway.java:238)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
[2020-12-02 15:51:17,195] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at py4j.GatewayConnection.run(GatewayConnection.java:238)
[2020-12-02 15:51:17,195] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,195] {{spark_submit_hook.py:436}} INFO - at java.lang.Thread.run(Thread.java:748)
[2020-12-02 15:51:17,195] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,195] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:51:17,214] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,214] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO ShutdownHookManager: Shutdown hook called
[2020-12-02 15:51:17,214] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,214] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO ShutdownHookManager: Deleting directory /tmp/spark-5dd50760-7c6a-4173-83d1-944b6d7d7858
[2020-12-02 15:51:17,215] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,215] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO ShutdownHookManager: Deleting directory /tmp/spark-d9d497e5-4635-4b2c-b057-ba01ca569db7
[2020-12-02 15:51:17,546] {{taskinstance.py:1088}} ERROR - Cannot execute: ['spark-submit', '--master', 'spark://spark:7077', '--conf', 'spark.master=spark://spark:7077', '--name', 'Spark Hello World', '--verbose', '--queue', 'root.default', '/usr/local/spark/app/hello-world.py', '/usr/local/spark/resources/data/airflow.cfg']. Error code is: 1.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 955, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/spark_submit_operator.py", line 181, in execute
self._hook.submit(self._application)
File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py", line 362, in submit
spark_submit_cmd, returncode
airflow.exceptions.AirflowException: Cannot execute: ['spark-submit', '--master', 'spark://spark:7077', '--conf', 'spark.master=spark://spark:7077', '--name', 'Spark Hello World', '--verbose', '--queue', 'root.default', '/usr/local/spark/app/hello-world.py', '/usr/local/spark/resources/data/airflow.cfg']. Error code is: 1.
[2020-12-02 15:51:17,547] {{taskinstance.py:1117}} INFO - All retries failed; marking task as FAILED
[2020-12-02 15:51:20,567] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:20,566] {{local_task_job.py:103}} INFO - Task exited with return code 1

[doglasfc-docker]

I got exactly same issue. Anyone please help to fix it? thanks!

[cordon-thiago]

In this merge #9 I made all images to work with the same spark version (3.0.1) to avoid this problem.