holdenk/spark-testing-base

Incompatibility with Spark 3.0.1

gnsantos opened this issue · 0 comments

The documentation points out that in order to have the testing framework be compatible with Spark 3, all that was necessary was to include "com.holdenkarau" %% "spark-testing-base" % "3.0.0_0.14.0" % "test" in the build.sbt file, however this package is not present in the Maven Central Repo, resulting in a failure to resolve dependencies and compile.

Furthermore, trying to use the latest stable release present on Maven along with Spark 3 result in the failure of tests that previously passed. Root cause seems to be:

java.lang.VerifyError: class org.apache.spark.sql.catalyst.plans.logical.UpdateAction overrides final method children.()Lscala/collection/Seq;

Attempts to bump fastxml dependecies yielded no change, and reverting back to Spark 2.4.1 made the tests pass again, which suggests that the issue is an incompatibility with Spark 3, despite #325 being merged.