build in support for 3.4 and dbr 12.2 and 13.x starter
Closed this issue · 11 comments
12.2 merges https://issues.apache.org/jira/browse/SPARK-41049 into its 3.3.2 base removing Stateful.
3.4 also changes bind in hofs, compilation handling, type checking and error message format (change between rc1 and rc4).
Frameless no longer works with the build.
Another compat layer needed.
frameless issue, occurs in the CodeGenTests using more complex encoding. Not sure if it's 12.2 relevant yet.
java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.ScalaReflection$.dataTypeFor(Lscala/reflect/api/TypeTags$TypeTag;)Lorg/apache/spark/sql/types/DataType;
at frameless.TypedEncoder$$anon$19.toCatalyst(TypedEncoder.scala:453)
at frameless.RecordEncoder.$anonfun$toCatalyst$2(RecordEncoder.scala:155)
1785768 seems to work for 12.2 lts - only failed on verifyInputParams - so possibly old 340 syntax is being used.
there will be an rc5 and confirmed the above frameless issue is present for dbr. So something extra 41045? was backported.
Raised typelevel/frameless#698 for the frameless fix
per typelevel/frameless#701 integrating with Spark Connect is probably impossible. Moreover the only expressions allowed in columns are those defined here. So there is no way to inject dq rules or functions through connect either - a bit of dead functionality.
dcb4ba1 uses normal 3.4
as 13.0 dbr is out this should also be tested against. Running 12.2 dbr against it only had the frameless failure (dataTypeFor). As such no special build will be created.
213 failures on 13.1 beta on latest snapshot (vs. the 1 on all other dbrs including 13.0). Individually running the tests c+p'ing into the notebook works. Likely a class loading issue as the base build / shades aren't the same - does raise concern on having 12.2/13.0 vs 13.1 builds numbering is off.
scalatest was it, can't now run individual tests but only 8 fail on 13.1 (7 new):
BaseFunctionalityTest
testComparableResults - all comparable_maps tests fail due to:
NoSuchMethodErrror org.apache.spark.sql.types.AtomicType.ordering: scala.math.Ordering
so a signature change not present on 13.0. Probably the same for the ValidationTests as well
ValidationTest
testFunctionSyntaxError is broken: 219
testOutputFunctionSyntaxError: 214
43019 is the reason for the ordering issue. 13.1 backports it (also declared in the release notes)
3.4 done, 12.2 all tests working, 13.1 as well