Delta Standalone: Jar Hell - Hadoop dependencies

Question

Delta Standalone: Jar Hell - Hadoop dependencies

VladimirPchelko opened this issue a year ago · 8 comments

package size becomes too large due to Hadoop dependencies

Answer 1 · 2023-04-20T18:17:03.000Z

@VladimirPchelko The package itself is not large. Do you mean when you build an uber jar that includes all jars, it's too large? Unfortunately, there is no great solution for this. We need Hadoop to access the cloud storage that's using Hadoop FileSystem APIs to provide the access.

Answer 2 · 2023-04-25T06:49:37.000Z

due to hadoop dependencies it is very problematic to use your project in AWS lambdas (needed uber jar) - without complicated manipulations, the packet size exceeds the limits

Answer 3 · 2023-04-28T15:09:32.000Z

Unfortunately, there is little we can do. We need Hadoop to access the cloud storage that's using Hadoop FileSystem APIs. I found this StackOverflow link that may be able to use a larger jar: https://stackoverflow.com/a/72646550/1038826 Could you check if it helps?

Answer 4 · 2023-04-30T14:06:39.000Z

I think to use a different approach, build a separate maven project, which will contain only really used libraries from hadoop (for example - exclude aws-sdk-bom (hadoop-aws), include aws-sdk-s3) пт, 28 апр. 2023 г. в 18:09, Shixiong Zhu ***@***.***>:

…

Unfortunately, there is little we can do. We need Hadoop to access the cloud storage that's using Hadoop FileSystem APIs. I found this StackOverflow link that may be able to use a larger jar: https://stackoverflow.com/a/72646550/1038826 Could you check if it helps? — Reply to this email directly, view it on GitHub <#541 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIJIVSZH7NJB5ACHRUXHQB3XDPMTPANCNFSM6AAAAAAXDTNJFY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Answer 5 · 2023-05-04T18:40:53.000Z

@VladimirPchelko - did this new approach work for you?

Answer 6 · 2023-05-05T10:03:31.000Z

Yes, it works. At first, it allows me to exclude aws-java-sdk-bom - it greatly reduces the size of uber jar. Many things like yarn, kafka, ,zookeeper, mapreduce, etc. are not needed for the functionality of Hadoop filesystems (S3 and DBFS). чт, 4 мая 2023 г. в 21:41, Scott Sandre ***@***.***>:

…

@VladimirPchelko <https://github.com/VladimirPchelko> - did this new approach work for you? — Reply to this email directly, view it on GitHub <#541 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIJIVS6J5O32QSL5XF53TSTXEPZ4BANCNFSM6AAAAAAXDTNJFY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Answer 7 · 2023-05-20T11:41:26.000Z

Would it not be better not to rely on the Hadoop libraries for cloud storage access, since all cloud storage providers have pure APIs in multiple languages. As an example, delta-rs, the Rust implementation of Deltalake is a good example where they've done this cleanly.

For example, we turned to Deltalake (and cloud storage) to move completely away from Hadoop, but now we find again having to depend on Hadoop libraries in order to use Deltalake, which to my mind doesn't make much sense.

Answer 8 · 2023-07-11T17:58:49.000Z

This repo has been deprecated and the code is moved under connectors module in https://github.com/delta-io/delta repository. Please create the issue in repository https://github.com/delta-io/delta. See #556 for details.