Task fail without logs
vinicvaz opened this issue · 2 comments
Hey guys!
I'm newbie to spark and I'm trying to setup a job I have in Glue to run distributed in multiple Lambdas.
I tried to run the sample code you provided for iceberg sample-accommodations-to-iceberg.py building the image using PYSPARK_VERSION=3.2.0 and commenting out the part for writing, just left the read csv part, but the lambda execution is failing without logs.
The lambda_handler was executed, the script file was downloaded but after the job submit the lambda task fail without any logs.
This is the log I'm getting:
In the cloud watch I see the same thing.
Can someone help me?
Thanks!
Sure, Just to confirm, It is just Iceberg or the Delta table script failing?
Looks like the time out setting in AWS Lambda is 3 sec please increase that to atleast 3 min