java.lang.NoClassDefFoundError: Could not initialize class org.jpmml.evaluator.FieldUtil

[shipengAlan]

Hi, I'm stuck with a runtime issue like this

java.lang.NoClassDefFoundError: Could not initialize class org.jpmml.evaluator.FieldUtil
    at org.jpmml.evaluator.InputFieldUtil$1.getDataType(InputFieldUtil.java:499)
    at org.jpmml.evaluator.FieldValue.create(FieldValue.java:377)
    at org.jpmml.evaluator.FieldValueUtil.create(FieldValueUtil.java:72)
    at org.jpmml.evaluator.InputFieldUtil.createInputValue(InputFieldUtil.java:479)
    at org.jpmml.evaluator.InputFieldUtil.prepareScalarInputValue(InputFieldUtil.java:130)
    at org.jpmml.evaluator.InputFieldUtil.prepareInputValue(InputFieldUtil.java:111)
    at org.jpmml.evaluator.InputField.prepare(InputField.java:73)

My spark project is based on scala 2.12 and sbt 1.4.7x.
Main dependency version:
spark-core: 3.1.1
jpmml-evaluator: 1.5.9

Do you know whether root cause of above issue is because of pmml-model/guava dependency conflict? And how to solve it?
I found in the spark-mllib library, there has the dependency to org.sparkproject.jpmml.model and org.sparkproject.dmg.pmml.
Thanks a lot

[vruusmann]

java.lang.NoClassDefFoundError: Could not initialize class org.jpmml.evaluator.FieldUtil

I assume that your classpath is set up correctly, in a sense that the o.j.e.FieldUtil class is physically present and successfully loadable.

If so, then it must be the case that JVM cannot load the o.j.e.FieldUtil class, because its class initializer (aka static initializer) fails.

I don't see anything fail-able in this class initializer code block:
https://github.com/jpmml/jpmml-evaluator/blob/1.5.9/pmml-evaluator/src/main/java/org/jpmml/evaluator/FieldUtil.java#L173-L195

I found in the spark-mllib library, there has the dependency to org.sparkproject.jpmml.model and org.sparkproject.dmg.pmml.

This observation relates to SPARK-15526. You're running Apache Spark 3.X, and all JPMML classes appear to be correctly isolated between Apache Spark Core (ie. org.sparkproject.jpmml.model.*) and JPMML-Evaluator (ie. org.jpmml.model).

Main dependency version:
spark-core: 3.1.1
jpmml-evaluator: 1.5.9

Do you know whether root cause of above issue is because of pmml-model/guava dependency conflict?

JPMML-Evaluator should work with any Guava version in range 19.X -- 31.X (and possibly newer).

You should check which kind of Guava dependency is included/available in your Apache Spark 3(.1) application environment. Is it a "plain" one, or is it a "shaded" one. In the latter case, you should import Guava yourself.

TLDR: Double-check your Apache Spark runtime classpath, and your build configuration. The JPMML-Evaluator classpath (in its default configuration) is definitely correct. There is something interfering with it, and it can't be investigated/fixed by me.

[vruusmann]

Another classpath debugging idea: write a small Java application that only tries to load a o.j.e.FieldUtil class, and see if the resulting exception stack trace is more informative - perhaps it reveals the identity of the missing 3rd party class (most likely some Guava class) that causes the class initializer to fail.

public class Main {

  static
  public void main(String[] args) throws Exception {
    Class<?> clazz = org.jpmml.evaluator.FieldUtil.class;
    System.out.println(clazz.getName());
  }
}

[shipengAlan]

[info]   +-org.jpmml:pmml-evaluator:1.5.9
[info]   | +-com.google.guava:guava:30.1-jre
[info]   | | +-com.google.code.findbugs:jsr305:3.0.2
[info]   | | +-com.google.errorprone:error_prone_annotations:2.3.4
[info]   | | +-com.google.guava:failureaccess:1.0.1
[info]   | | +-com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
[info]   | | +-com.google.j2objc:j2objc-annotations:1.3
[info]   | | +-org.checkerframework:checker-qual:3.5.0
[info]   | | 
[info]   | +-org.apache.commons:commons-math3:3.4.1
[info]   | +-org.apache.commons:commons-math3:[3.1, 3.6.1] (evicted by: 3.4.1)
[info]   | +-org.jpmml:pmml-model:1.5.9
[info]   |   +-org.jpmml:pmml-agent:1.5.9

The guava is 30.1-jre
Not sure whether it is due to "com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava"
I can try your debug idea. Thank you

[vruusmann]

The guava is 30.1-jre

The current Guava dependency declaration is here:
https://github.com/jpmml/jpmml-evaluator/blob/1.6.4/pom.xml#L152-L188

It should be safe to exclude all Guava transitive dependencies, except for the com.google.guava:failureaccess dependency.

I can try your debug idea.

The o.j.e.FieldUtil class initializer refers to the o.j.e.CacheUtil utility class. You may alternatively try loading that class first (perhaps there is a wrong Guava Cache builder specification in your application's system properties?).

Anyway, my observation is that the class loading fails before the actual PMML content is touched. That is, the failure is not model-dependent, it must be application config-dependent.