Hudi Job on Lambda - Directory /var/task/metastore_db cannot be created.

[sha12br]

Hi Team,

I have been trying to run Hudi job on Lambda, the function is getting properly invoked and the files are read fine as well. But the issue seems to be while writing i guess where its trying to create derby.log under /var/task (which is readonly).

I have followed through this repo guide, from CloudWatch Logs :

ERROR XBM0H: Directory /var/task/metastore_db cannot be created.

I have tried changing the config for "spark.sql.warehouse.dir" to local and S3 as well, Still for some reason its trying to create under /var/task (where the spark-submit was invoked).

Could someone please help me out here. Any suggestions/help would be great.

Thanks.