linkedin/isolation-forest

Facing issues with json4s package, while saving model. Also not able to create a fat jar due to version conflict between liberaries.

Closed this issue · 8 comments

sandeep1288 commented

I downloaded the iForest jar from : https://dl.bintray.com/linkedin/maven/com/linkedin/isolation-forest/isolation-forest_2.11/0.2.2/
But while trying to save model, I'm getting below options. I tried multiple versions of json4s modules, but still not able to fix the error.
java.lang.NoSuchMethodError: org.json4s.jackson.JsonMethods$.parse(Lorg/json4s/JsonInput;Z)Lorg/json4s/JsonAST$JValue;

Apart from this I tried to compile iForest using the mentioned steps, but that didn't work, and threw version conflict errors.
Is it possible for you to share the detailed info, which've been used to compile the package, and versions of modules being used.

sandeep1288 commented

Below function threw error:
private def getMetadataToSave(
instance: Params,
spark: SparkSession,
extraMetadata: Option[JObject] = None): String = {

  val uid = instance.uid
  val cls = instance.getClass.getName
  val params = instance.extractParamMap.toSeq
  val jsonParams = render(params.map { case ParamPair(p, v) =>
    p.name -> parse(p.jsonEncode(v))
  }.toList)
  val basicMetadata = ("class" -> cls) ~
    ("timestamp" -> System.currentTimeMillis()) ~
    ("sparkVersion" -> spark.sparkContext.version) ~
    ("uid" -> uid) ~
    ("paramMap" -> jsonParams)
  val metadata = extraMetadata match {
    case Some(jObject) => basicMetadata ~ jObject
    case None => basicMetadata
  }
  val metadataJson = compact(render(metadata))

  metadataJson
}

}

And problem is happening while saving model.

jverbus commented

What version of Spark and Scala are you using?

The recommended versions for the project are:

Scala 2.11.8
Spark 2.3.0

These are the versions used to produce the jars on Bintray.

sandeep1288 commented

I'm using databrick platform, and below is the scala and spark version:
spark-version : 2.4.3
scala-version : 2.11.12

In order to make current iForest working on latest spark version, what are the essential steps which I can follow. There should not be much difference between spark 2.3 and 2.4, apart from json4s.

jverbus commented

It is currently recommended to use Scala 2.11 and Spark 2.3.

You will need to build the isolation-forest library yourself using Spark 2.4 if you want to use it in a Spark 2.4 environment. You will need to change some dependencies in the module-level build.gradle file.

Please try the instructions in this ticket: #1

sandeep1288 commented

Thanks a lot. I'll give it a try.

jverbus commented

@sandeep1288: Cool. Please let me know if it works. Thanks!

sandeep1288 commented

Thanks a lot, I'm able to make it working with latest versions of spark.

  • 👍1
sandeep1288 commented