submit jar on lambda failed

Question

submit jar on lambda failed

tyyzqmf opened this issue 6 months ago · 4 comments

We try to submit jar on lambda, and modify the spark_submit function:

subprocess.run([
            "spark-submit", 
            "--class", "software.aws.solution.xxxx.DataProcessor",
            "--master", "local[*]",
            "--conf", "spark.driver.bindAddress=127.0.0.1",
            "--conf", "spark.driver.memory=2g",
            "--conf", "spark.executor.memory=2g",
            "--conf", "spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
            "--files", "/tmp/GeoLite2-City.mmdb",
            "/tmp/spark-etl-1.1.0.jar", 
            ], check=True, env=os.environ)

Then get error logs:

24/03/06 02:47:09 INFO Utils: Successfully started service 'SparkUI' on port 4040.
--
24/03/06 02:47:09 INFO SparkContext: Added JAR file:/tmp/spark-etl-1.1.0.jar at spark://169.254.53.45:43149/jars/spark-etl-1.1.0.jar with timestamp 1709693228782
24/03/06 02:47:09 INFO SparkContext: Added file file:///tmp/GeoLite2-City.mmdb at file:///tmp/GeoLite2-City.mmdb with timestamp 1709693228782
24/03/06 02:47:09 INFO Utils: Copying /tmp/GeoLite2-City.mmdb to /tmp/spark-d34b112a-2839-4fab-8494-7a4507d6dd6e/userFiles-71d13aa6-257e-4cb0-bd05-0955aa29d8ab/GeoLite2-City.mmdb
24/03/06 02:47:10 INFO Executor: Starting executor ID driver on host 169.254.53.45
24/03/06 02:47:10 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
24/03/06 02:47:10 INFO Executor: Fetching file:///tmp/GeoLite2-City.mmdb with timestamp 1709693228782
24/03/06 02:47:10 INFO Utils: /tmp/GeoLite2-City.mmdb has been previously copied to /tmp/spark-d34b112a-2839-4fab-8494-7a4507d6dd6e/userFiles-71d13aa6-257e-4cb0-bd05-0955aa29d8ab/GeoLite2-City.mmdb
24/03/06 02:47:10 INFO Executor: Fetching spark://169.254.53.45:43149/jars/spark-etl-1.1.0.jar with timestamp 1709693228782
24/03/06 02:47:10 ERROR Utils: Aborting task
java.io.IOException: Failed to connect to /169.254.53.45:43149
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:288)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:230)
at org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:399)
at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$openChannel$4(NettyRpcEnv.scala:367)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:366)
at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:799)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:557)
at org.apache.spark.executor.Executor.$anonfun$updateDependencies$13(Executor.scala:1010)
at org.apache.spark.executor.Executor.$anonfun$updateDependencies$13$adapted(Executor.scala:1002)
at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:1002)
at org.apache.spark.executor.Executor.<init>(Executor.scala:273)
at org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:64)
at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:132)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:222)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:585)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:953)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:947)
at software.aws.solution.clickstream.DataProcessor.runWithSpark(DataProcessor.java:115)
at software.aws.solution.clickstream.DataProcessor.main(DataProcessor.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /169.254.53.45:43149
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
24/03/06 02:47:10 ERROR SparkContext: Error initializing SparkContext.
java.io.IOException: Failed to connect to /169.254.53.45:43149
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:288)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:230)
at org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:399)
at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$openChannel$4(NettyRpcEnv.scala:367)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:366)
at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:799)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:557)
at org.apache.spark.executor.Executor.$anonfun$updateDependencies$13(Executor.scala:1010)
at org.apache.spark.executor.Executor.$anonfun$updateDependencies$13$adapted(Executor.scala:1002)
at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:1002)
at org.apache.spark.executor.Executor.<init>(Executor.scala:273)
at org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:64)
at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:132)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:222)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:585)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:953)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:947)
at software.aws.solution.clickstream.DataProcessor.runWithSpark(DataProcessor.java:115)
at software.aws.solution.clickstream.DataProcessor.main(DataProcessor.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /169.254.53.45:43149
Caused by: java.net.ConnectException: Connection refused

Answer 1 · 2024-03-06T08:23:30.000Z

solved by add ~~spark.driver.bindAddress=127.0.0.1~~ spark.driver.host=127.0.0.1

Answer 2 · 2024-05-02T11:09:16.000Z

@tyyzqmf i see from the issue description, you already have set this config.
Do you need to set this elsewhere ?

I’m facing a similar issue where I have the conf set to bind address to local but still getting the error

thanks

Answer 3 · 2024-05-02T12:45:10.000Z

@tyyzqmf i see from the issue description, you already have set this config. Do you need to set this elsewhere ?

I’m facing a similar issue where I have the conf set to bind address to local but still getting the error

thanks

My mistake. I have been solving this problem by adding a new config:
spark.driver.host=127.0.0.1

Answer 4 · 2024-05-02T17:31:46.000Z

@tyyzqmf i see from the issue description, you already have set this config. Do you need to set this elsewhere ?
I’m facing a similar issue where I have the conf set to bind address to local but still getting the error
thanks

My mistake. I have been solving this problem by adding a new config: spark.driver.host=127.0.0.1

Thank you
Also setting this spark env variable also is working for me: SPARK_LOCAL_IP="127.0.0.1"