Include S3 based Shuffle storage plugin

Question

Include S3 based Shuffle storage plugin

gauravtanwar03 opened this issue 7 months ago · 3 comments

AWS lambda has a limit of 1024 open File descriptors that leads to task result loss failure if you are merging data in the target tables for building a idempotent ingestion pipelines.

Solution: Use S3 based shuffle storage plugin that can act as a replacement for disk in AWS lambda.

Answer 1 · 2024-03-04T15:57:04.000Z

@gauravtanwar03 Good thinking. Do you have a prototype ready?