sbt-dependency-submission resolving wrong versions?

Question

sbt-dependency-submission resolving wrong versions?

mdedetrich opened this issue a year ago · 6 comments

At Pekko we have added sbt-dependency-submission to our project (see apache/pekko#366) however we are getting an interesting/odd bug where although the plugin is submitting the dependencies, the version of sbt dependencies appear to be off?

Incase you can't see the dependabot alerts, here is an example

The thing is, the project is currently using Jackson 2.14.3 (see https://github.com/apache/incubator-pekko/blob/main/project/Dependencies.scala#L37-L38) and if you checkout the project and run libraryDependencies in the sbt shell

[IJ]pekko > libraryDependencies
[info] pki / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, com.hierynomus:asn-one:0.5.0, org.slf4j:slf4j-api:1.7.36, org.scalatest:scalatest:3.1.4:test)
[info] actor-testkit-typed / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, ch.qos.logback:logback-classic:1.2.11:optional;provided;test, junit:junit:4.13.2:optional;provided;test, org.scalatest:scalatest:3.1.4:optional;provided;test, org.scalatestplus:junit-4-13:3.1.4.0:test)
[info] coordination / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test)
[info] cluster-sharding-typed / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10)
[info] actor-typed / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.slf4j:slf4j-api:1.7.36)
[info] docs / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.apache.pekko:pekko-theme-paradox:0.0.0+38-68da3106-SNAPSHOT:paradox-theme, org.scalatest:scalatest:3.1.4:test, junit:junit:4.13.2:test, io.spray:spray-json:1.3.6:test, com.google.code.gson:gson:2.9.1:test, org.iq80.leveldb:leveldb:0.12:optional;provided)
[info] actor-typed-tests / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10)
[info] testkit / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test, io.dropwizard.metrics:metrics-core:4.2.10:test, io.dropwizard.metrics:metrics-jvm:4.2.10:test, org.latencyutils:LatencyUtils:2.0.3:test, org.hdrhistogram:HdrHistogram:2.1.12:test)
[info] stream / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.reactivestreams:reactive-streams:1.0.4, com.typesafe:ssl-config-core:0.4.3, org.scalatest:scalatest:3.1.4:test)
[info] multi-node-testkit / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, io.netty:netty:3.10.6.Final)
[info] persistence-query / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:test, junit:junit:4.13.2:test, commons-io:commons-io:2.11.0:test, org.iq80.leveldb:leveldb:0.12:optional;provided, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided)
[info] osgi / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.osgi:org.osgi.core:6.0.0, org.osgi:org.osgi.compendium:5.0.0, ch.qos.logback:logback-classic:1.2.11:test, commons-io:commons-io:2.11.0:test, com.googlecode.pojosr:de.kalpatec.pojosr.framework:0.2.1:test, org.ops4j.pax.tinybundles:tinybundles:3.0.0:test, org.scalatest:scalatest:3.1.4:test, junit:junit:4.13.2:test)
[info] persistence-typed-tests / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:test, ch.qos.logback:logback-classic:1.2.11:test)
[info] persistence-testkit / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:test, ch.qos.logback:logback-classic:1.2.11:test)
[info] persistence / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.iq80.leveldb:leveldb:0.12:optional;provided, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided, org.scalatest:scalatest:3.1.4:test, org.scalatestplus:junit-4-13:3.1.4.0:test, junit:junit:4.13.2:test, commons-io:commons-io:2.11.0:test, commons-codec:commons-codec:1.15:test)
[info] cluster-tools / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test)
[info] remote-tests / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test, io.netty:netty:3.10.6.Final, io.aeron:aeron-driver:1.38.1, io.aeron:aeron-client:1.38.1)
[info] protobuf-v3 / libraryDependencies
[info] 	List(com.google.protobuf:protobuf-java:3.16.1:optional;provided)
[info] stream-tests-tck / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:test, org.scalatestplus:testng-6-7:3.1.4.0:test, org.scalatestplus:scalacheck-1-14:3.1.4.0:test, junit:junit:4.13.2:test, org.reactivestreams:reactive-streams-tck:1.0.4:test)
[info] protobuf / libraryDependencies
[info] 	List()
[info] actor / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, com.typesafe:config:1.4.2)
[info] discovery / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test)
[info] bench-jmh / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.openjdk.jmh:jmh-core:1.32, org.openjdk.jmh:jmh-generator-bytecode:1.32, org.openjdk.jmh:jmh-generator-reflection:1.32, ch.qos.logback:logback-classic:1.2.11, org.iq80.leveldb:leveldb:0.12:optional;provided, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided, org.jctools:jctools-core:3.3.0)
[info] cluster-metrics / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, io.kamon:sigar-loader:1.6.6-rev002:optional;provided;test, org.slf4j:jul-to-slf4j:1.7.36:test, org.slf4j:log4j-over-slf4j:1.7.36:test, ch.qos.logback:logback-classic:1.2.11:test, org.scalatestplus:mockito-3-4:3.1.4.0:test)
[info] bill-of-materials / libraryDependencies
[info] 	List()
[info] stream-tests / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:test, org.scalatestplus:scalacheck-1-14:3.1.4.0:test, junit:junit:4.13.2:test, commons-io:commons-io:2.11.0:test, com.google.jimfs:jimfs:1.1:test)
[info] stream-testkit / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:test, org.scalatestplus:scalacheck-1-14:3.1.4.0:test, junit:junit:4.13.2:test)
[info] remote / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.agrona:agrona:1.15.1, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test, com.google.jimfs:jimfs:1.1:test, com.google.protobuf:protobuf-java:3.16.1:test, io.netty:netty:3.10.6.Final:optional, io.aeron:aeron-driver:1.38.1:optional, io.aeron:aeron-client:1.38.1:optional)
[info] distributed-data / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.lmdbjava:lmdbjava:0.7.0, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test)
[info] cluster / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test)
[info] actor-tests / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test, org.scalatestplus:junit-4-13:3.1.4.0:test, org.scalatestplus:scalacheck-1-14:3.1.4.0:test, commons-codec:commons-codec:1.15:test, org.apache.commons:commons-math:2.2:test, com.google.jimfs:jimfs:1.1:test, com.spotify:docker-client:8.16.0:test, com.sun.activation:javax.activation:1.2.0:provided;test)
[info] slf4j / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.slf4j:slf4j-api:1.7.36, ch.qos.logback:logback-classic:1.2.11:test)
[info] cluster-typed / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10)
[info] serialization-jackson / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, com.fasterxml.jackson.core:jackson-core:2.14.3, com.fasterxml.jackson.core:jackson-annotations:2.14.3, com.fasterxml.jackson.core:jackson-databind:2.14.3, com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.14.3, com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.14.3, com.fasterxml.jackson.module:jackson-module-parameter-names:2.14.3, com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.14.3, com.fasterxml.jackson.module:jackson-module-scala:2.14.3, org.lz4:lz4-java:1.8.0, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test)
[info] persistence-tck / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:compile, junit:junit:4.13.2:compile, org.iq80.leveldb:leveldb:0.12:optional;provided, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided)
[info] stream-typed / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10)
[info] persistence-shared / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.iq80.leveldb:leveldb:0.12:optional;provided, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided, ch.qos.logback:logback-classic:1.2.11:test)
[info] persistence-typed / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.iq80.leveldb:leveldb:0.12:optional;provided, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided, ch.qos.logback:logback-classic:1.2.11:test)
[info] cluster-sharding / libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10, org.iq80.leveldb:leveldb:0.12:optional;provided;multi-jvm;test, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test, commons-io:commons-io:2.11.0:test, site.ycsb:core:0.17.0:test)
[info] libraryDependencies
[info] 	List(org.scala-lang:scala-library:2.13.10)
[IJ]pekko >

You can see that all of the jackson versions are being resolved to 2.14.3

Answer 1 · 2023-06-12T09:43:10.000Z

sbt-dependenyc-submission does not submit only the runtime dependencies but also the dependencies of your build (sbt plugins) and development configurations (Test, ScalaTools, ScalaDocTools). That's why your snapshot contains three dependencies to jackson-databind:

2.14.3 is coming from the Compile dependencies.
2.13.3 is coming from the ScalaDocTools dependencies. It is a transitive dependency of scaladoc_3.

sbt:pekko> ++3.3.0
sbt:pekko> show update
...
[info] 	scala-doc-tool:
...
[info] 		com.fasterxml.jackson.core:jackson-databind:2.13.3:default
...

and 2.9.10 is coming from

sbt:pekko> reload plugins
sbt:pekko> show dependencyTree
...
[info]   +-com.github.sbt:sbt-unidoc:0.5.0
[info]   +-com.hpe.sbt:sbt-pull-request-validator:1.0.0
[info]   | +-org.kohsuke:github-api:1.93
[info]   |   +-com.fasterxml.jackson.core:jackson-databind:2.9.2
...

You should be able to fix the dependabot alert by adding libraryDependency += "com.fasterxml.jackson.core:jackson-databind:2.14.3" in your project/plugins.sbt.

Answer 2 · 2023-06-12T09:45:50.000Z

As a side note, if you want to ignore the dependencies coming from some configurations you can use the configs-ignore input.

There is no way currently to ignore the dependencies of the build itself though.

Answer 3 · 2023-06-12T10:41:08.000Z

sbt-dependenyc-submission does not submit only the runtime dependencies but also the dependencies of your build (sbt plugins) and development configurations (Test, ScalaTools, ScalaDocTools). That's why your snapshot contains three dependencies to jackson-databind

Oh wow, I didn't realize it also grabbed dependencies from sbt plugin itself, I though it was just grabbing dependencies from runtime/compile. Thanks for letting me know, I presume to filter out sbt-plugin dependencies one would use scala-tool and scala-doc-tool?

Answer 4 · 2023-06-12T11:25:04.000Z

Thanks for letting me know, I presume to filter out sbt-plugin dependencies one would use scala-tool and scala-doc-tool?

scala-tool contains the compiler and its dependencies
scala-doc-tool contains scaladoc and its dependencies
The sbt-plugin dependencies are the compile dependencies of the meta-build. There is no way currently to ignore dependencies for the meta-build.

Answer 5 · 2023-06-12T11:54:10.000Z

Actually I am wrong: we don't include the meta-build dependencies in the snapshot.

The jackson-databind:2.9.8 comes from the Test dependencies of actor-tests, coordination and discovery:

pekko > show actor-tests/Test/dependencyTree
[info] org.apache.pekko:pekko-actor-tests_2.13:0.0.0+26699-495dc110+20230612-1344-SNA..
...
[info]   +-com.spotify:docker-client:8.16.0
...
[info]   | +-org.glassfish.jersey.media:jersey-media-json-jackson:2.22.2
...
[info]   | | +-com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:2.9.8
...
[info]   | | | +-com.fasterxml.jackson.core:jackson-databind:2.9.8
...
[info]   | | +-com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.9.8
...
[info]   | | |   +-com.fasterxml.jackson.core:jackson-databind:2.9.8
...

You should be able to override this dependency in your build. Or you can ignore some projects/configs with the modules-ignore or configs-ignore inputs.

Answer 6 · 2023-06-12T12:25:30.000Z

Thanks for the help, found the root culprit https://github.com/spotify/docker-client