spark 3.4 support - replacing dataTypeFor logic
chris-twiner opened this issue · 8 comments
ScalaReflection.dataTypeFor has been completely removed in 3.4 rc4 (and seemingly in Databricks 12.2 as well) assumed as part of SPARK-41045. MapGroups also has a change but I've not yet tracked that down.
adding
spark_34_rc4: https://repository.apache.org/content/repositories/orgapachespark-1438/
to a ~/.sbt/repositories above maven-central is enough to reproduce along with this changed sbt.
I'll attempt to verify if just adding the previous impl back is sufficient next week but I'll happily look at other alternatives.
Commit that removes the function: apache/spark@030c1ba#diff-aa90bc536778701618f741339d52ce249bb50f808b138e96a481f2b31b462b65L69-L116
(Leaving it here so we have more context)
MapGroups is a change from after 3.3.0 introducing order from SPARK-38591
managed to sneak in some time over lunch - copying the small bits of impl over works fine against the test suite and Quality's tests where I detected it. The branch on my fork has the impl, but clearly the MapGroups constructor needs a work around for cross version and although an empty ordering "works" I've no idea if it's meant to.
I'll not easily be able to test on databricks until there is a published snapshot, but checking the MapGroups constructor shows 5 params e.g. MapGroups(null, null, null, null, null)(null, null, null) compiles but 4 params doesn't and shows the dataOrder param. There may be other dragons there but at least these two are consistent with 3.4.
I'll see if I can get a nice cross compile support for MapGroups next week.
per 1186e5b I've added changes to manage the MapGroups across versions - verified against 3.4 RC5. Also tested with Quality locally.
Per the rc5 vote it's likely we'll see at least another RC. I'll update and test again against them as they drop.
Should I raise a pull now, knowing it'll only fail until after 3.4.0 is officially published?
Hey @chris-twiner why not? If an open PR hanging around is not bothering you, it could be a nice reminder for us.
Awesome progress you've done btw 🎉 I'm super happy to see it.
more than welcome, #699 is raised for it.