Harden dependencies

Question

Harden dependencies

dvallant opened this issue 3 years ago · 21 comments

Sometimes when I build my own software project, which uses docker client, the build fails unexpectedly. I found out that the reason for this is the use of version ranges in dependency management. This makes it impossible to reproduce builds. So my question is: would it be possible to harden them?

Answer 1 · 2022-02-11T09:54:53.000Z

There are tradeoffs and I can certainly try to declare the preferred dependency versions and release a strict pom.xml/modules.json with an improved dependency management. Could you tell me whether there are specific transitive dependencies, which make your builds fail? I would focus on thouse first.

Then, out of curiosity: is your project open sourced? You might have noticed that I'm migrating away from Groovy to Kotlin and a more type safe model for the docker client. I would keep an eye on your project to not continuously break it. My personal usage is mainly driven by the https://github.com/gesellix/gradle-docker-plugin, which doesn't use the complete docker client api. So, knowing about another project would help me getting a clearer picture.

Answer 2 · 2022-02-14T11:11:23.000Z

First of all, thank you for investing time in this issue.

Could you tell me whether there are specific transitive dependencies, which make your builds fail?

Currently i needed to harden the following dependencies to get my project working as expected:
com.kohlschutter.junixsocket:junixsocket-core:pom:2.4.0
com.squareup.okio:okio:jar:2.5.0
com.squareup.okhttp3:okhttp:jar:5.0.0-alpha.3

is your project open sourced?

No, sorry. It's a project developed for my company that we only use internally.

Answer 3 · 2022-02-24T20:30:04.000Z

I'm working on a cleanup of the dependency configuration. Before going full circle with an official release I'd like to ask if you're able to change your setup to use the GitHub Package Registry in addition to Maven Central. Would that theoretically work for you?

Please note that I'm also going to upgrade to Kotlin 1.6, which shouldn't actually break anything, because I'm not using any Kotlin 1.6 specific features, yet. So, this is only a heads-up about the changed transitive dependencies.

Answer 4 · 2022-04-23T15:15:28.000Z

@dvallant could you check if the current release works better for you regarding transitive dependencies? We made some dependency versions more explicit while keeping an indication of compatible version ranges. This should work with Maven as well as Gradle.
Please leave some feedback here, thanks!

Answer 5 · 2022-04-25T09:45:32.000Z

Maven complains about a missing dependency to 'com.kohlschutter.junixsocket:junixsocket-core:jar:2.4.0':
Could not find artifact com.kohlschutter.junixsocket:junixsocket-core:jar:2.4.0

I have to exclude junixsocket-core and define it myself:

            <dependency>
                <groupId>com.kohlschutter.junixsocket</groupId>
                <artifactId>junixsocket-core</artifactId>
                <version>2.4.0</version>
                <type>pom</type>
            </dependency>

Answer 6 · 2022-04-25T20:04:03.000Z

Thanks for the quick response!

~~Can you explain why or whether the <type>pom</type> is necessary?~~
Found it myself, it's the actual artifact type for junixsocket-core.

Answer 7 · 2022-04-26T12:57:59.000Z

I also have problems with the dependency on "com.squareup.okio:okio". The defined version, 3.1.0, is scoped with "runtime" which is ok when it's only needed at runtime but it's ignored by maven packaging (self contained jar), which caused me to define it myself. But the downloaded jar doesn't contain any classes, just a structure unknown to me:

commonMain/
hashFunctions/
META INF/
nativeMain/
nonJvmMain/
unixMain/

Could you please explain what i'm doing wrong?

Answer 8 · 2022-04-26T16:51:52.000Z

I'll need to look into that myself, as a first thought I suspect that okio is published in several artifacts for different platforms. Gradle uses a different model than Maven, so there might be some unsupported situations in Maven, while Gradle selects different artifacts for the id "com.squareup.okio:okio". If that's the case, I'll certainly find a way to publish a Maven-compatible pom.xml, similar to the fix for junixsocket-core@pom.

Answer 9 · 2022-04-26T21:02:41.000Z

This one looks related to the okio-in-Maven issue: square/okio#1067

Answer 10 · 2022-04-27T07:25:21.000Z

This one looks related to the okio-in-Maven issue: square/okio#1067

Thanks for this hint. After adding 'okio-jvm' as a runtime dependency it works :-)

Answer 11 · 2022-05-05T19:40:37.000Z

With the latest release 2022-05-05T20-38-00 everything should work with Maven out of the box and less custom dependency fixes on your side. I noticed some duplicated transitive dependencies, though, so please keep an eye out and consider using Maven's dependency enforcement plugin.

I finally made a minimal example project at https://github.com/gesellix/docker-client-examples with both Maven and Gradle projects. Feel free to file a pull request there, if you'd like to see more advanced examples matching your use case. Maybe I still missed some relevant details.

Thanks so much for your patience and your support! Please leave a note here with feedback if everything's working fine now.

Answer 12 · 2022-05-09T17:28:46.000Z

Ping @dvallant in case you didn't see my previous message. Your feedback is appreciated :)

Answer 13 · 2022-05-09T17:36:26.000Z

@gesellix Sorry, at the moment I am very busy but I will try to give a feedback later this week :-)

Answer 14 · 2022-05-09T17:45:16.000Z

@dvallant alright, no pressure, and thanks for the quick update!

Answer 15 · 2022-05-16T13:35:21.000Z

@gesellix Sorry for the late response. Now I had time to try the new release. Before, I had to define following dependencies:

<dependency>
    <groupId>com.kohlschutter.junixsocket</groupId>
    <artifactId>junixsocket-core</artifactId>
    <version>2.4.0</version>
    <type>pom</type>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>com.squareup.okio</groupId> 
    <artifactId>okio-jvm</artifactId>
    <version>3.1.0</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>org.jetbrains.kotlin</groupId>
    <artifactId>kotlin-stdlib-common</artifactId>
    <version>1.6.21</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>org.jetbrains.kotlin</groupId>
    <artifactId>kotlin-reflect</artifactId>
    <version>1.6.21</version>
    <scope>runtime</scope>
</dependency>

Now the only dependency I need is:

<dependency>
    <groupId>com.kohlschutter.junixsocket</groupId>
    <artifactId>junixsocket-core</artifactId>
    <version>2.4.0</version>
    <type>pom</type>
    <scope>runtime</scope>
</dependency>

This is really much better :-)

But since the update I face an issue with the validation of the interval field for healthchecks:
"message":"rpc error: code = InvalidArgument desc = ContainerSpec: Interval in HealthConfig cannot be less than 1ms"}

I will do more testing and if necessary file an issue in docker-compose-v3.

Answer 16 · 2022-05-16T17:20:30.000Z

Thanks for leaving feedback!

I'm a bit surprised about the junixsocket-core dependency still being required by you to be added. I'll try to fix that one, too :)
Regarding the Interval error: that one might not be caused by something in docker-compose-v3, but in the remote-api dependency. See docker-remote-api for details. If you're willing to fix that one (maybe by setting a default), please also consider the sibling project docker-remote-api-client, because that's where the actual behaviour should be implemented. If you need any more hints, please feel free to leave a note here or file a more specific issue at one of the mentioned repositories. Thanks!

Answer 17 · 2022-05-17T12:14:15.000Z

I'm a bit surprised about the junixsocket-core dependency still being required by you to be added. I'll try to fix that one, too :)

I was surprised too :-) So I did a little bit of further investigation and voilá: Another library in my project has a dependency to junixsocket but to an older version. After excluding the old one everything works as expect :-D Thank you very much!

ad Interval error: After some research I think the problem is the datatype. It should be Long not Int. I will file an issue in docker-remote-api.

Answer 18 · 2022-05-17T17:13:46.000Z

voilá

Now that's great news! 😌

Interval data type

You're right, that's another area I'm already working on (pulling changes from the moby/upstream swagger config). So, an issue would be perfect, no need on your end to create a pull request.

Thanks for keeping me updated, this is very helpful!

I suppose this issue is resolved, so I'm going to close it in a few days, giving you a chance to veto if I missed anything ;)

Answer 19 · 2022-05-18T06:10:23.000Z

no need on your end to create a pull request

Too late ;-) docker-client/docker-remote-api#71

issue is resolved

Yes it is. Thank you very much for investing your time in this issue.

Answer 20 · 2022-05-18T06:48:29.000Z

Thanks! :)

Answer 21 · 2022-06-27T21:34:08.000Z

So, we're going to go back to the official okio artifact (instead of okio-jvm), see square/okio#1125 where okio got a fix specifically for Maven users. We're not going to rush things, because going back to plain okio might cuase issues with dynamic version ranges. There's also square/okhttp#7351, which should also be released before we go on and change our dependencies.

This is just a heads-up :)