Can't connect to docker using TLS and Docker credentials

Question

Can't connect to docker using TLS and Docker credentials

samrocketman opened this issue 9 years ago · 11 comments

samrocketman commented 9 years ago

Server configuration

CentOS 7
Docker version 1.8.2-el7.centos, build a01dc02/1.8.2
Placed certificates in /etc/docker/. ca.pem (ca cert), cert.pem (server cert), key.pem (server private key).
Added to /etc/sysconfig/docker with OPTIONS="--tlsverify -H tcp://127.0.0.1:443"

Client configuration

Same system as server.
I've created a client directory, let's say /path/docker, and have added my ca.pem (ca cert), cert.pem (client cert), and key.pem (client private key) to said path.
My jenkins is listening on localhost:8080.

I had to dig for it but found the docker cloud authentication configuration instructions #245 (comment).

Jenkins specs

Jenkins ver. 1.646
docker-plugin 0.16.0
Added Docker Certificates Directory credentials and configured path to /path/docker (where client {ca,cert,key}.pem are located).
Added a docker cloud with Docker URL pointed to https://127.0.0.1:443 and using aforementioned credentials.

Issue

I can't get the docker plugin to properly connect. I click the test connection button and it throws an error. When I test with curl it succeeds with my client certificates.

cd /path/docker
#successful auth test
curl --cert ./cert.pem --key ./key.pem --cacert ./ca.pem https://127.0.0.1:443/info
#successfully denied access
curl -k https://127.0.0.1:443/info

Here's a stack trace from the web UI when I use the Test Connection button in the docker cloud settings of the configure page.

Error in SSL Configuration

com.github.dockerjava.api.DockerClientException: Error in SSL Configuration
    at com.github.dockerjava.jaxrs.DockerCmdExecFactoryImpl.init(DockerCmdExecFactoryImpl.java:140)
    at com.github.dockerjava.core.DockerClientImpl.withDockerCmdExecFactory(DockerClientImpl.java:131)
    at com.github.dockerjava.core.DockerClientBuilder.build(DockerClientBuilder.java:73)
    at com.nirima.jenkins.plugins.docker.client.ClientBuilderForPlugin.build(ClientBuilderForPlugin.java:65)
    at com.nirima.jenkins.plugins.docker.DockerCloud$DescriptorImpl.doTestConnection(DockerCloud.java:615)
    at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
    at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:123)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.github.dockerjava.api.DockerClientException: org.bouncycastle.asn1.pkcs.PrivateKeyInfo cannot be cast to org.bouncycastle.openssl.PEMKeyPair
    at com.github.dockerjava.core.LocalDirectorySSLConfig.getSSLContext(LocalDirectorySSLConfig.java:57)
    at com.github.dockerjava.jaxrs.DockerCmdExecFactoryImpl.init(DockerCmdExecFactoryImpl.java:138)
    ... 61 more
Caused by: java.lang.ClassCastException: org.bouncycastle.asn1.pkcs.PrivateKeyInfo cannot be cast to org.bouncycastle.openssl.PEMKeyPair
    at com.github.dockerjava.core.CertificateUtils.loadPrivateKey(CertificateUtils.java:127)
    at com.github.dockerjava.core.CertificateUtils.createKeyStore(CertificateUtils.java:45)
    at com.github.dockerjava.core.LocalDirectorySSLConfig.getSSLContext(LocalDirectorySSLConfig.java:50)
    ... 62 more

Please note my SSL certs are x509 ASCII PEM format. Any idea what could be going wrong?

Workaround

This section is an edit of the original issue. I noted a workaround below.

Answer 1 · 2016-01-27T21:00:15.000Z

@magnayn I'm able to reproduce the stack trace from the Jenkins script console.

//variables
String serverUrl = "https://127.0.0.1:443"
String credentialsId = "606ed7a0-0cab-4c76-bd57-a9f4aea9a692"
String version = "1.8"
Integer readTimeout = 0
Integer connectTimeout = 0

//import com.nirima.jenkins.plugins.docker.DockerCloud$DescriptorImpl
//def x = Jenkins.instance.getExtensionList(com.nirima.jenkins.plugins.docker.DockerCloud$DescriptorImpl)[0]
//println x.metaClass.methods*.name.sort().unique()
//x.doTestConnection(serverUrl, credentialsId, version, readTimeout, connectTimeout)

import com.github.dockerjava.core.DockerClientConfig
import com.nirima.jenkins.plugins.docker.client.DockerCmdExecConfig
import com.github.dockerjava.api.DockerClient
import com.github.dockerjava.api.model.Version
import com.nirima.jenkins.plugins.docker.client.ClientConfigBuilderForPlugin
import com.nirima.jenkins.plugins.docker.client.DockerCmdExecConfigBuilderForPlugin
import com.nirima.jenkins.plugins.docker.client.ClientBuilderForPlugin


final DockerClientConfig clientConfig = ClientConfigBuilderForPlugin.dockerClientConfig()
    .forServer(serverUrl, version)
    .withCredentials(credentialsId)
    .build()

final DockerCmdExecConfig execConfig = DockerCmdExecConfigBuilderForPlugin.builder()
    .withReadTimeout(readTimeout)
    .withConnectTimeout(connectTimeout)
    .build()

DockerClient dc = ClientBuilderForPlugin.builder()
    .withDockerClientConfig(clientConfig)
    .withDockerCmdExecConfig(execConfig)
    .build()

Version verResult = dc.versionCmd().exec()
"Version = " + verResult.getVersion()

It's basically an excerpt of doTestConnection(). The exception is called when the following is executed:

DockerClient dc = ClientBuilderForPlugin.builder()
    .withDockerClientConfig(clientConfig)
    .withDockerCmdExecConfig(execConfig)
    .build()

Answer 2 · 2016-01-27T22:24:02.000Z

Upgraded to Docker version 1.9.1, build a34a1d5. curl still works and Jenkins config does not. It doesn't appear to reach a step of connecting to docker. It seems to fail on setup before a connection is ever made to docker.

Answer 3 · 2016-01-27T22:57:55.000Z

Here's some example docker certificates if you want to test what I am doing. Please note, even when I remove the PEM info from the certs it still doesn't work and consistently fails with the same error.

Answer 4 · 2016-01-27T23:23:17.000Z

The root of the exception occurs here. I've checked my 4096-bit private key and it seems OK.

$ openssl rsa -in key.pem -check -noout
RSA key ok

Answer 5 · 2016-01-27T23:31:51.000Z

It FINALLY worked because I had to change how the private key was presented.

mv key.pem key.bak
openssl rsa -in key.bak -text > key.pem

Apparently, it only takes the private key successfully if you give it an extended -text format. The wiki needs to be updated with how to use TLS auth. I'll get around to it if nobody does.

Answer 6 · 2016-01-28T00:09:01.000Z

Filed a bug report with upstream library.

Answer 7 · 2016-02-13T10:35:58.000Z

Since docker-java/docker-java#441 is closed and the maintainer says they can't reproduce with their library using a different plugin then it narrows it to this plugin.

Answer 8 · 2016-03-17T04:57:05.000Z

As noted in docker-java/docker-java#441 there seems to be an open PR which could potentially fix this issue. I'm looking forward to it.

See the fix at docker-java/docker-java#502

Answer 9 · 2016-03-24T01:11:56.000Z

Hey @samrocketman I just wanted to thank you for discovering the problem and workaround, helped me through my day! Thank you!

Answer 10 · 2016-03-25T01:35:09.000Z

Awesome @jquast :D glad my experience could help you. I went ahead and updated the original issue so the workaround is pointed out right away.

Answer 11 · 2017-10-10T13:13:19.000Z

Docker plugin 0.17 now uses docker-commons for credentials management, which should avoid such issues making clearer definition about the expected keys format.