Add support for Scala 2.13
benedeki opened this issue · 6 comments
Background
It has been requested to be able to use the library in a project built for on Scala 2.13
AFAIK hats and AbsaOSS/spark-hofs#19 are for Spark 2.
Spark 2 does not support Scala 2.13.
Does it make sense to update these to use scala 2.13?
Hofs was created to add functionality missing in Spark 2. But Spark 3 has 'transform' etc. So hofs might become obsolete.
But hats adds nice interface on the top of standard functions for better array manipulation. And if it can be rewritten to use standard Spark's transform(), it makes sense to still have it in Spark 3.
Hofs was created to add functionality missing in Spark 2. But Spark 3 has 'transform' etc. So hofs might become obsolete.
But
hatsadds nice interface on the top of standard functions for better array manipulation. And if it can be rewritten to use standard Spark'stransform(), it makes sense to still have it in Spark 3.
Ideal to be done as part of #33
It would be cool, but would require the release to be split to version for Spark 2 and another one for Spark 3 (as spark-commons, and spark-partition-sizing have it)
I'm going to go for the same implementation for all Scala versions for now. Optimizing hofs away from Scala 2.12 and Scala 2.13 will be another issue and another PR.
I'll try to remove hofs dependency from Scala 2.12 and Scala 2.13 builds.