Setup for Java env
rsbCodeZ opened this issue · 2 comments
SoAL framework approach was replicated in Java environment, we have spark task bundled in jar file which was invoked via SparkLauncher class as part of SoAL.
This worked fine when tested on AWS lambda emulator provided
Docker base image was ->
FROM public.ecr.aws/lambda/java:11
But in AWS lambda environment below error is encountered -
/opt/spark/bin/spark-class: line 93: /dev/fd/62: No such file or directory
/opt/spark/bin/spark-class: line 93: /dev/fd/62: No such file or directory
/opt/spark/bin/spark-class: line 97: CMD: bad array subscript
The above error prevents the launch of spark task.
This error dev/fd/62: No such file or directory was encountered when default spark-class as part of spark installation was used as part of lambda execution.
Upon configuring custom spark-class as below and copying spark-class file to spark home the SparkLauncher triggered Spark job jar successfully
spark-class
java_path=/var/lang/bin/java
exec $java_path -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx2g "$@"
Docker file changes to include spark-class in SPARK_HOME similar to spark-on-aws-lambda sample code
COPY spark-class $SPARK_HOME/bin/
RUN chmod -R 755 $SPARK_HOME