Spark 3.0
Closed this issue · 4 comments
Would it be possible to add support for Spark 3.0 ? We upgrading our project to Spark 3.0.1 and Atum fails with the following error on disableControlMeasuresTracking
even though enableControlMeasuresTracking
was called before.
Control framework tracking is not initialized.
java.lang.IllegalStateException: Control framework tracking is not initialized.
at za.co.absa.atum.core.Atum$.preventNotInitialized(Atum.scala:229)
at za.co.absa.atum.core.Atum$.preventNotInitialized(Atum.scala:233)
at za.co.absa.atum.core.Atum$.dispose(Atum.scala:163)
at za.co.absa.atum.AtumImplicits$SparkSessionWrapper.disableControlMeasuresTracking(AtumImplicits.scala:144)
Hello @pokpetr, the plan is to support spark3 in the near future. I cannot give you a precise date now but will keep you posted.
As for this issue.
- Can we get more info about the code or maybe an example?
- Is it something that worked with 2.4 and not is not working?
Sorry, I was wrong. The original issue is not spark3 incompatibility but Scala 2.12 incompatibility.
SparkSessionWrapper.enableControlMeasuresTracking
throws the following error
Caused by: java.lang.NoClassDefFoundError: scala/Product$class
The "Control framework tracking is not initialized" error is caused by calling disableControlMeasuresTracking
on the spark session where the tracking isn't initialized because of the error above.
Anyway I assume that the upgrade to Scala 2.12 is not possible without upgrade to spark3.
Are you using Atum from Maven central directly? Currently, Atum on maven is built only with/for scala 2.11 (2.11 doesn't work with spark3). You can build it yourself with 2.12 and see if it works. Otherwise please wait for the new release.
Hello @pokpetr, please try the newly released Atum
<dependency>
<groupId>za.co.absa</groupId>
<artifactId>atum_2.12</artifactId>
<version>3.2.0</version>
</dependency>