AWScala requires a version of jackson-databind incompatible with spark 1.6

[ludflu]

Thanks for AWScala, its great!

Unfortunately, I needed to use S3 in conjunction with AWS EMR/Spark, and I ran into a dependency problem.

When I include AWScala in my spark project, ("com.github.seratch" %% "awscala" % "0.5.+") I get the following JsonMappingException:

com.fasterxml.jackson.databind.JsonMappingException: Could not find creator property with name 'id' (in class org.apache.spark.rdd.RDDOperationScope)

It seems that Spark is expecting "com.fasterxml.jackson.core" % "jackson-databind" % "2.4.4" which I think is a dependency of awsJavaSdkVersion = "1.10.59" ?

I tried excluding jackson-databind 2.4.4. as described here:

http://stackoverflow.com/questions/31039367/spark-parallelize-could-not-find-creator-property-with-name-id

But then AWScala broke:

java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper;
    at com.amazonaws.internal.config.InternalConfig.<clinit>(InternalConfig.java:43)
    at com.amazonaws.internal.config.InternalConfig$Factory.<clinit>(InternalConfig.java:304)
    at com.amazonaws.util.VersionInfoUtils.userAgent(VersionInfoUtils.java:139)
    at com.amazonaws.util.VersionInfoUtils.initializeUserAgent(VersionInfoUtils.java:134)
    at com.amazonaws.util.VersionInfoUtils.getUserAgent(VersionInfoUtils.java:95)
    at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:53)
    at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:416)
    at awscala.s3.S3Client.<init>(S3.scala:287)
    at awscala.s3.S3$.apply(S3.scala:15)
    at awscala.s3.S3$.at(S3.scala:19)

So I rewrite my code to use the aws-java-sdk directly, specifying version 1.10.64
(which was not only more verbose, but also less safe what with all the null checking java nonsense)

[alexarchambault]

👍