error starting build pod java.io.IOException: Pipe not connected

Question

error starting build pod java.io.IOException: Pipe not connected

rawlingsj opened this issue 7 years ago · 9 comments

Seems to happen when there's low resources or multiple jobs running. This has been seen on OSO and GKE.

Executing shell script inside container [maven] of pod [kubernetes-137ebf2065f949d4acac4e019ed07af7-1e96524904d1e]
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

java.io.IOException: Pipe not connected
	at java.io.PipedOutputStream.write(PipedOutputStream.java:140)
	at java.io.OutputStream.write(OutputStream.java:75)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:125)
	at hudson.Launcher$ProcStarter.start(Launcher.java:384)
	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:157)
	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:63)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:172)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
	at groovy.lang.GroovyObject$invokeMethod.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)
	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:115)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:16)
	at stageProject.call(/var/jenkins_home/jobs/fabric8-cd/jobs/fabric8-maven-plugin/branches/master/builds/16/libs/github.com/fabric8io/fabric8-pipeline-library/vars/stageProject.groovy:18)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:74)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE

Answer 1 · 2017-04-18T14:47:02.000Z

this is a dump of the recent jenkins server logs

Excess workload after pending Spot instances: 1
Apr 18, 2017 2:30:29 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Apr 18, 2017 2:30:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
Created Pod: kubernetes-137ebf2065f949d4acac4e019ed07af7-1e96524904d1e
Apr 18, 2017 2:30:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
Waiting for Pod to be scheduled (0/100): kubernetes-137ebf2065f949d4acac4e019ed07af7-1e96524904d1e
Apr 18, 2017 2:30:31 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
Accepted connection #140 from /10.16.3.16:54268
Apr 18, 2017 2:30:39 PM INFO hudson.slaves.NodeProvisioner$2 run
Kubernetes Pod Template provisioning successfully completed. We have now 4 computer(s)
Apr 18, 2017 2:33:12 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.45 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:12 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:12 PM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent
Received PUSH for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:12 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.40 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:12 PM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent
Received PUSH for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.40 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:33:29 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Apr 18, 2017 2:33:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
Created Pod: kubernetes-d1e751c682cf44258527ee001a5a84cc-1e98f0cf330c6
Apr 18, 2017 2:33:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
Waiting for Pod to be scheduled (0/100): kubernetes-d1e751c682cf44258527ee001a5a84cc-1e98f0cf330c6
Apr 18, 2017 2:33:30 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
Accepted connection #141 from /10.16.2.192:54688
Apr 18, 2017 2:33:33 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:33 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:33 PM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent
Received PUSH for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:33 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.45 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:33 PM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent
Received PUSH for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.45 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:36 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:36 PM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent
Received PUSH for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:36 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.40 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:36 PM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent
Received PUSH for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.40 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:39 PM INFO hudson.slaves.NodeProvisioner$2 run
Kubernetes Pod Template provisioning successfully completed. We have now 5 computer(s)
Apr 18, 2017 2:33:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:33:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:33:55 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:55 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:56 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:56 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.41 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:56 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:56 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.41 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:56 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.45 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:33:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:34:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:34:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:34:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:34:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:34:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:34:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:34:39 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:34:39 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:34:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:34:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:34:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:34:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:35:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:35:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:35:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:35:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:35:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:35:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:35:39 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:35:39 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:35:42 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminating Kubernetes instance for slave kubernetes-137ebf2065f949d4acac4e019ed07af7-1e96524904d1e
Apr 18, 2017 2:35:42 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminated Kubernetes instance for slave kubernetes-137ebf2065f949d4acac4e019ed07af7-1e96524904d1e
Apr 18, 2017 2:35:42 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Disconnected computer kubernetes-137ebf2065f949d4acac4e019ed07af7-1e96524904d1e
Apr 18, 2017 2:35:42 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
fabric8-cd/fabric8-maven-plugin/master #16 completed: FAILURE
Apr 18, 2017 2:35:42 PM WARNING hudson.model.listeners.RunListener report
RunListener failed
java.util.ConcurrentModificationException
	at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
	at java.util.TreeMap$KeyIterator.next(TreeMap.java:1265)
	at io.fabric8.kubernetes.pipeline.devops.elasticsearch.BuildListener.createBuild(BuildListener.java:71)
	at io.fabric8.kubernetes.pipeline.devops.elasticsearch.BuildListener.onCompleted(BuildListener.java:49)
	at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:211)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:618)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1400(WorkflowRun.java:125)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:901)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1092)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:400)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Apr 18, 2017 2:35:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:35:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:35:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:35:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:36:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:36:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:36:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:36:19 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Apr 18, 2017 2:36:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
Created Pod: kubernetes-e9706a5ff8074c468e1a35dc3516c250-1e9b6a1af6191
Apr 18, 2017 2:36:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
Waiting for Pod to be scheduled (0/100): kubernetes-e9706a5ff8074c468e1a35dc3516c250-1e9b6a1af6191
Apr 18, 2017 2:36:20 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
Accepted connection #142 from /10.16.3.17:34658
Apr 18, 2017 2:36:29 PM INFO hudson.slaves.NodeProvisioner$2 run
Kubernetes Pod Template provisioning successfully completed. We have now 5 computer(s)
Apr 18, 2017 2:38:35 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminating Kubernetes instance for slave kubernetes-d1e751c682cf44258527ee001a5a84cc-1e98f0cf330c6
Apr 18, 2017 2:38:35 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminated Kubernetes instance for slave kubernetes-d1e751c682cf44258527ee001a5a84cc-1e98f0cf330c6
Apr 18, 2017 2:38:35 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Disconnected computer kubernetes-d1e751c682cf44258527ee001a5a84cc-1e98f0cf330c6
Apr 18, 2017 2:38:35 PM WARNING jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
Computer.threadPoolForRemoting [#627] for kubernetes-d1e751c682cf44258527ee001a5a84cc-1e98f0cf330c6 terminated
java.io.EOFException
	at org.jenkinsci.remoting.nio.NioChannelHub$3.run(NioChannelHub.java:617)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Apr 18, 2017 2:38:35 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
fabric8-ui/fabric8-runtime-console/PR-311 #2 completed: FAILURE
Apr 18, 2017 2:38:35 PM WARNING hudson.model.listeners.RunListener report
RunListener failed
java.util.ConcurrentModificationException
	at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
	at java.util.TreeMap$KeyIterator.next(TreeMap.java:1265)
	at io.fabric8.kubernetes.pipeline.devops.elasticsearch.BuildListener.createBuild(BuildListener.java:71)
	at io.fabric8.kubernetes.pipeline.devops.elasticsearch.BuildListener.onCompleted(BuildListener.java:49)
	at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:211)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:618)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1400(WorkflowRun.java:125)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:901)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1092)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:400)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Apr 18, 2017 2:41:27 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminating Kubernetes instance for slave kubernetes-e9706a5ff8074c468e1a35dc3516c250-1e9b6a1af6191
Apr 18, 2017 2:41:27 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminated Kubernetes instance for slave kubernetes-e9706a5ff8074c468e1a35dc3516c250-1e9b6a1af6191
Apr 18, 2017 2:41:27 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Disconnected computer kubernetes-e9706a5ff8074c468e1a35dc3516c250-1e9b6a1af6191
Apr 18, 2017 2:41:27 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
fabric8-ui/fabric8-runtime-console/master #146 completed: FAILURE
Apr 18, 2017 2:41:27 PM WARNING hudson.model.listeners.RunListener report
RunListener failed
java.util.ConcurrentModificationException
	at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
	at java.util.TreeMap$KeyIterator.next(TreeMap.java:1265)
	at io.fabric8.kubernetes.pipeline.devops.elasticsearch.BuildListener.createBuild(BuildListener.java:71)
	at io.fabric8.kubernetes.pipeline.devops.elasticsearch.BuildListener.onCompleted(BuildListener.java:49)
	at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:211)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:618)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1400(WorkflowRun.java:125)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:901)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1092)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:400)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Answer 2 · 2017-04-18T18:09:46.000Z

@iocanel I don't suppose this means anything to you does it? We've started running lots of PR builds and we can have 5 running concurrently on a three node GKE cluster, which has plenty of space resources. Roughly every 12 hours we start seeing these errors and I can only bounce the jenkins master to recover.

Any clues how to investigate what's going on?

Answer 3 · 2017-04-18T18:24:33.000Z

Interesting comment from

We have been having this issue, and did an investigation. It seems to be failing on pipe write in the first "cd" command to get the workspace. From what we understand the underlying kubernetes-client library does not expose a stable connection. From what we understand the latch was implemented wrongly (?? We think). The latch is initialized once in the container "start" method, but a new connection is opened for each sh step. This causes the "waitQuietly" to pass through straight away without waiting, on all sh connections but the first. Once we fixed this the "OnOpen" callback in the websocket is never called, indicating the connection is not made, and the latch was only passed on thread interrupt.

https://issues.jenkins-ci.org/browse/JENKINS-40825?focusedCommentId=290693&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-290693

Answer 4 · 2017-07-11T11:19:17.000Z

I faced this issue on AWS also.
got working after increasing resources and bouncing the jenkins pod.

Answer 5 · 2017-07-25T10:46:46.000Z

@vikasjio could you elaborate? How are your resources set now to resolve the issue?

Answer 6 · 2017-07-25T11:43:47.000Z

Hi, For me, things were working initially; but started failing once I increased loads/applications in my cluster. So, I just added 2 more VMs in my cluster, which already had 4 worker VMs of the same size. From: tettaji [mailto:notifications@github.com] Sent: 25 July 2017 16:17 To: fabric8io/fabric8-pipeline-library Cc: Vikas Chander; Mention Subject: Re: [fabric8io/fabric8-pipeline-library] error starting build pod java.io.IOException: Pipe not connected (#206) @vikasjio<https://github.com/vikasjio> could you elaborate? How are your resources set now to resolve the issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#206 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AURkBK8Ht_1GM4eVykNXtxmKLBQwHBHwks5sRceYgaJpZM4NAcbE>. "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

Answer 7 · 2017-08-22T08:27:51.000Z

This is fixed in the kubernetes-plugin jenkinsci/kubernetes-plugin#182, just need to release a new Jenkins image with the fix included and update the pipeline library to work with 0.13-SNAPSHOT as there seems to be some breaking change with the way env vars are added to pods. I will hopefully have this done in the next day or two.

Answer 8 · 2018-03-19T13:56:22.000Z

@rawlingsj is this fixed in the newest release ?

just experienced it on my setup:
Jenkins: Core 2.32.3
Fabric8-pipeline-library: 2.2.311
Plugins:

	Kubernetes	1.3.2	0.10-SNAPSHOT (private-70e60d4e-jamesrawlings)
	Kubernetes :: Pipeline :: DevOps Steps	1.5	1.4-SNAPSHOT (private-e8baa5c6-jamesrawlings)
	Kubernetes :: Pipeline :: Kubernetes Steps	1.5	1.4-SNAPSHOT (private-e8baa5c6-jamesrawlings)

Answer 9 · 2018-03-19T13:59:03.000Z

@marvinosswald I'm not involved in the fabric8 project anymore I'm afraid so not sure where things are. From memory however I think the latest kubernetes-plugin did include a number if fixes to avoid this so I think it should be fixed.