Posting in Mattermost not working

Hi,

in my environment Jenkins and Mattermost (GitLab Omnibus) are running on two different servers (CentOS, OpenJDK 8, no Docker) with self signed certificates. I added the certificates to the JKS and the appropriate Mattermost, Jenkins and OS folders on both servers.

Jenkins Settings:

Endpoint: https://mymattermost.local:8066/hooks/123xyz
Channel: jenkins-testing
Icon: empty
Build Server URL: https://myjenkins.local

When I run the connection test in the Jenkins log shows the following:

May 23, 2018 1:13:39 PM INFO jenkins.plugins.mattermost.StandardMattermostService publish
Posting: to jenkins-testing@https://mymattermost.local:8066/hooks/123xyz: Mattermost/Jenkins plugin: you're all set! (parameters: endpoint='https://mymattermost.local:8066/hooks/123xyz', room='jenkins-testing', icon='', buildServerUrl='https://myjenkins.local/') (good)
May 23, 2018 1:13:39 PM INFO org.apache.commons.httpclient.HttpMethodBase readResponseBody
Response content length is not known
May 23, 2018 1:13:39 PM INFO org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
I/O exception (javax.net.ssl.SSLPeerUnverifiedException) caught when processing request: peer not authenticated
May 23, 2018 1:13:39 PM INFO org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Retrying request
May 23, 2018 1:13:39 PM INFO org.apache.commons.httpclient.HttpMethodBase readResponseBody
Response content length is not known
May 23, 2018 1:13:39 PM INFO org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
I/O exception (javax.net.ssl.SSLPeerUnverifiedException) caught when processing request: peer not authenticated
May 23, 2018 1:13:39 PM INFO org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Retrying request
May 23, 2018 1:13:39 PM INFO org.apache.commons.httpclient.HttpMethodBase readResponseBody
Response content length is not known
May 23, 2018 1:13:39 PM INFO org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
I/O exception (javax.net.ssl.SSLPeerUnverifiedException) caught when processing request: peer not authenticated
May 23, 2018 1:13:39 PM INFO org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Retrying request
May 23, 2018 1:13:39 PM INFO org.apache.commons.httpclient.HttpMethodBase readResponseBody
Response content length is not known
May 23, 2018 1:13:39 PM WARNING jenkins.plugins.mattermost.StandardMattermostService publish
Error posting to Mattermost
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
	at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:431)
	at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.verifyHostName(SSLProtocolSocketFactory.java:257)
	at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:200)
	at org.apache.commons.httpclient.HttpConnection.tunnelCreated(HttpConnection.java:793)
	at org.apache.commons.httpclient.HttpMethodDirector.executeConnect(HttpMethodDirector.java:521)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:178)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:404)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:330)
	at jenkins.plugins.mattermost.StandardMattermostService.publish(StandardMattermostService.java:99)
	at jenkins.plugins.mattermost.StandardMattermostService.publish(StandardMattermostService.java:41)
	at jenkins.plugins.mattermost.MattermostNotifier$DescriptorImpl.doTestConnection(MattermostNotifier.java:452)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
	at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:198)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:199)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.jenkinsci.plugins.cas.spring.security.CasSingleSignOutFilter.doFilter(CasSingleSignOutFilter.java:39)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:564)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
	at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
	at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
May 23, 2018 1:13:39 PM INFO jenkins.plugins.mattermost.StandardMattermostService publish
Posting succeeded

The gitlab_mattermost_access log shows nothing in this case.

Testing from Jenkins server using curl works as expected:
curl -i -X POST -d 'payload={"text": "Hello\nText"}' https://mymattermost.local:8066/hooks/123xyz
Here the gitlab_mattermost_access log shows "POST /hooks/123xyz HTTP/1.1" 200 2 "" "curl/7.58.0"

Any idea why the connection fails?
Thanks in advance!

did you look at #12 and #16 ? Both are related to SSL issues. The last one may very well be the solution - or not, but I'd appreciate you verify this first :-)

Speaking of #12 I use OpenJDK 1.8 so there souldn't be a problem. Additionally I verified that TLSv1 is activated.
The certificates (whole chain including root CA) are already imported to the Java and the Jenkins Keystores and copied to the following folders
/etc/pki/ca-trust/source/anchors/
/etc/pki/tls/certs/
$JENKINS_HOME/.keystore/
The webservers on both servers use the whole chain like mentioned in #16.

However I don't understand why the logs on the Mattermost server don't show any connection attempts.

They may not show any attempts if the SSL negotiation does not work...

Since this is a recurring issue, I looked for a more generic solution; maybe this is something to try?
https://support.cloudbees.com/hc/en-us/articles/217309497-Test-a-SSL-connection-from-Jenkins

I just wonder because every other connection between the two servers works as ecxpected.

Thanks for the link! I set up the pipeline and did some testing. There seem to be handshake problems with TLSv1.2. I will investigate further...

Did you find anything that still points at this plugin?

I will close this for inactivity