Scheduled build pods get stuck after completing a job
Mistobaan opened this issue · 2 comments
Mistobaan commented
I am able to run the jobs but it spins up a pod each time.
The pod is not then deleted or reused. After 10 executions and the 10 pod created the system start complain that it cannot run any more pods because the quota has been reached.
I am using this image as suggested previously:
https://hub.docker.com/r/dmarell/jenkins-k8s-slave/
The slave seems to die with this:
2017-08-19T23:04:46.756902655Z INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@7f4b5c9
2017-08-19T23:04:48.242115636Z Aug 19, 2017 11:04:48 PM hudson.remoting.jnlp.Main createEngine
2017-08-19T23:04:48.242171896Z INFO: Setting up slave: jnlp-p99vb
2017-08-19T23:04:48.246947683Z Aug 19, 2017 11:04:48 PM hudson.remoting.jnlp.Main$CuiListener <init>
2017-08-19T23:04:48.246972045Z INFO: Jenkins agent is running in headless mode.
2017-08-19T23:04:48.544674818Z Aug 19, 2017 11:04:48 PM hudson.remoting.jnlp.Main$CuiListener status
2017-08-19T23:04:48.544730294Z INFO: Locating server among [http://jenkins-ui.jenkins.svc.cluster.local:8080]
2017-08-19T23:04:48.689677815Z Aug 19, 2017 11:04:48 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
2017-08-19T23:04:48.689720643Z INFO: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
2017-08-19T23:04:48.698744558Z Aug 19, 2017 11:04:48 PM hudson.remoting.jnlp.Main$CuiListener status
2017-08-19T23:04:48.698772923Z INFO: Agent discovery successful
2017-08-19T23:04:48.698776729Z Agent address: jenkins-discovery.jenkins.svc.cluster.local
2017-08-19T23:04:48.698779425Z Agent port: 50000
2017-08-19T23:04:48.698782095Z Identity: 88:91:55:3a:b2:fe:0f:14:5c:d1:61:36:71:9b:4e:e1
2017-08-19T23:04:48.703344023Z Aug 19, 2017 11:04:48 PM hudson.remoting.jnlp.Main$CuiListener status
2017-08-19T23:04:48.703364229Z INFO: Handshaking
2017-08-19T23:04:48.70478802Z Aug 19, 2017 11:04:48 PM hudson.remoting.jnlp.Main$CuiListener status
2017-08-19T23:04:48.704800564Z INFO: Connecting to jenkins-discovery.jenkins.svc.cluster.local:50000
2017-08-19T23:04:48.731755365Z Aug 19, 2017 11:04:48 PM hudson.remoting.jnlp.Main$CuiListener status
2017-08-19T23:04:48.731804218Z INFO: Trying protocol: JNLP4-connect
2017-08-19T23:04:48.959515601Z Aug 19, 2017 11:04:48 PM hudson.remoting.jnlp.Main$CuiListener status
2017-08-19T23:04:48.959556353Z INFO: Remote identity confirmed: 88:91:55:3a:b2:fe:0f:14:5c:d1:61:36:71:9b:4e:e1
2017-08-19T23:04:48.973805798Z Aug 19, 2017 11:04:48 PM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
2017-08-19T23:04:48.973832468Z INFO: [JNLP4-connect connection to jenkins-discovery.jenkins.svc.cluster.local/10.59.252.68:50000] Local headers refused by remote: Unknown client name: jnlp-p99vb
2017-08-19T23:04:49.014159744Z Aug 19, 2017 11:04:48 PM hudson.remoting.jnlp.Main$CuiListener status
2017-08-19T23:04:49.014199861Z INFO: Protocol JNLP4-connect encountered an unexpected exception
2017-08-19T23:04:49.014204621Z java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: jnlp-p99vb
2017-08-19T23:04:49.01420811Z at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
2017-08-19T23:04:49.014211678Z at hudson.remoting.Engine.innerRun(Engine.java:385)
2017-08-19T23:04:49.014214408Z at hudson.remoting.Engine.run(Engine.java:287)
2017-08-19T23:04:49.014217159Z Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: jnlp-p99vb
2017-08-19T23:04:49.014228625Z at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:375)
2017-08-19T23:04:49.014231318Z at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:432)
2017-08-19T23:04:49.014233952Z at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
2017-08-19T23:04:49.014236357Z at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
2017-08-19T23:04:49.014238739Z at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
2017-08-19T23:04:49.014241274Z at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
2017-08-19T23:04:49.01424363Z at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
2017-08-19T23:04:49.014246185Z at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
2017-08-19T23:04:49.014248652Z at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
2017-08-19T23:04:49.01425085Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017-08-19T23:04:49.01425315Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-08-19T23:04:49.014255701Z at hudson.remoting.Engine$1$1.run(Engine.java:94)
2017-08-19T23:04:49.014258094Z at java.lang.Thread.run(Thread.java:748)
2017-08-19T23:04:49.014260137Z Suppressed: java.nio.channels.ClosedChannelException
2017-08-19T23:04:49.014262794Z ... 7 more
2017-08-19T23:04:49.014264841Z
2017-08-19T23:04:49.02400576Z Aug 19, 2017 11:04:49 PM hudson.remoting.jnlp.Main$CuiListener status
2017-08-19T23:04:49.024053647Z INFO: Connecting to jenkins-discovery.jenkins.svc.cluster.local:50000
2017-08-19T23:04:49.025735611Z Aug 19, 2017 11:04:49 PM hudson.remoting.jnlp.Main$CuiListener status
2017-08-19T23:04:49.025749014Z INFO: Server reports protocol JNLP4-plaintext not supported, skipping
2017-08-19T23:04:49.026269615Z Aug 19, 2017 11:04:49 PM hudson.remoting.jnlp.Main$CuiListener status
2017-08-19T23:04:49.026279338Z INFO: Protocol JNLP3-connect is not enabled, skipping
Mistobaan commented
After some more digging I found this JENKINS-41384 issue.
disabling the strict verification in the http://<master ip or name>/script did the trick.
jenkins.slaves.DefaultJnlpSlaveReceiver.disableStrictVerification=truerenjithkm86 commented
Upgrading Kubernetes plugin(1.30.11) to the latest fixed the same issue in Jenkins 2.289.3