Tunnel termination hanging when using Jenkins with ECS slaves
justinbarry opened this issue · 2 comments
justinbarry commented
See process output below:
[root@d72757b430d8 src]# ps -aux
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 1.5 1.0 12961964 331964 ? Ssl 09:38 0:36 java -Duser.home=/home/jenkins -cp /usr/share/jenkins/slave.jar hudson.remoting.jnlp.Main -headless -url http://10.4.1.5 f34bb1a77887fbb790ee7baff1abb643f9344c84d0e16fa0ae48dd67ea11ecfd jenkins-44700c09073e
root 426 0.0 0.0 12892 2768 ? S 09:42 0:00 sh -c echo $$ > '/home/jenkins/workspace/d-outcomes_HO-test-ie-in-cd-6GICDBXOWCXKWVWSICMEHJLKF4MHPTPJDJGILMJQHGBPNGY3VGQQ/jsclient@tmp/durable-f84387dd/pid'; jsc=durable-59651029b78f11e9a37f7b961cbdfc1d; JE
root 428 0.0 0.0 12892 2784 ? S 09:42 0:00 /bin/sh -xe /home/jenkins/workspace/d-outcomes_HO-test-ie-in-cd-6GICDBXOWCXKWVWSICMEHJLKF4MHPTPJDJGILMJQHGBPNGY3VGQQ/jsclient@tmp/durable-f84387dd/script.sh
root 429 0.0 0.1 1063648 49904 ? Sl 09:42 0:00 npm
root 439 0.0 0.0 12896 2800 ? S 09:42 0:00 sh -c set -e && node scripts/local.runner.js -c conf/parallel_local.conf.js -e chrome
root 440 0.0 0.1 1204000 42656 ? Sl 09:42 0:01 node scripts/local.runner.js -c conf/parallel_local.conf.js -e chrome
root 465 0.0 0.0 0 0 ? Zs 09:42 0:00 [BrowserStackLoc] <defunct>
root 487 0.0 0.1 938152 44372 ? Sl 09:43 0:00 /root/.browserstack/BrowserStackLocal --daemon stop --log-file /home/jenkins/workspace/d-outcomes_HO-test-ie-in-cd-6GICDBXOWCXKWVWSICMEHJLKF4MHPTPJDJGILMJQHGBPNGY3VGQQ/jsclient/local.log --key xoWQ57MU6qEkK
root 496 0.0 0.1 906392 42508 ? Sl 09:43 0:02 /root/.browserstack/BrowserStackLocal --daemon stop --log-file /home/jenkins/workspace/d-outcomes_HO-test-ie-in-cd-6GICDBXOWCXKWVWSICMEHJLKF4MHPTPJDJGILMJQHGBPNGY3VGQQ/jsclient/local.log --key xoWQ57MU6qEkK
root 519 0.4 0.0 13028 3076 ? Ss 10:17 0:00 /bin/bash
root 533 0.0 0.0 14904 2268 ? R+ 10:17 0:00 ps -aux
[root@d72757b430d8 src]#
Elastic Container Service](https://aws.amazon.com/ecs/) runs Docker containers leading me to think could be related to #25.
By default Jenkins uses set -xe per the docs . Is this enough to satisfy the intent of the workaround in #25?
Please let me know what further info I can provide to assist in debugging.
xscheiner commented
Also experiencing this. We run jenkins in a container with two slave containers. We recently built a local jenkins running on an osx machine with one slave container from the same dockerfile to try and reproduce the problem. We can't and we cannot see what is different. Manually killing the process works and the builds complete.
SlavikCA commented
We are having same issue.
More details:
here my config file:
[sfursov@qaauto01 webdriverio-browserstack2]$ cat conf/local.conf.js
var browserstack = require('browserstack-local');
exports.config = {
user: process.env.BROWSERSTACK_USERNAME || 'slavikfursov1',
key: process.env.BROWSERSTACK_ACCESS_KEY || '??????',
updateJob: true,
specs: [
'./tests/specs/local_test.js'
],
exclude: [],
capabilities: [{
browser: 'chrome',
name: 'local_test',
build: 'webdriver-browserstack',
'browserstack.local': true,
'browserstack.debug': true
}],
logLevel: 'debug',
coloredLogs: true,
screenshotPath: './errorShots/',
baseUrl: '',
waitforTimeout: 10000,
connectionRetryTimeout: 90000,
connectionRetryCount: 3,
host: 'hub.browserstack.com',
before: function () {
var chai = require('chai');
global.expect = chai.expect;
chai.Should();
},
framework: 'mocha',
mochaOpts: {
ui: 'bdd',
timeout: 60000
},
// Code to start browserstack local before start of test
onPrepare: function (config, capabilities) {
console.log("Connecting local");
return new Promise(function (resolve, reject) {
exports.bs_local = new browserstack.Local();
exports.bs_local.start({ 'key': exports.config.key, 'force-proxy':'', 'local-proxy-host':'****', 'local-proxy-port':'8080' }, function (error) {
if (error) return reject(error);
console.log('Connected. Now testing...');
resolve();
});
});
},
// Code to stop browserstack local after end of test
onComplete: function (exitCode, config, capabilities, results) {
exports.bs_local.stop();
},
}
Tests runs fine 1st time,
But on 2nd run, I'm getting:
/home/sfursov/webdriverio-browserstack2/node_modules/@wdio/cli/build/run.js:94
throw e;
^
LocalError: Either another browserstack local client is running on your machine or some server is listening on port 45691
at /home/sfursov/webdriverio-browserstack2/node_modules/browserstack-local/lib/Local.js:58:20
at ChildProcess.exithandler (child_process.js:285:7)
at ChildProcess.emit (events.js:189:13)
at ChildProcess.EventEmitter.emit (domain.js:441:20)
at maybeClose (internal/child_process.js:970:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
I found it caused by hanging processes:
[sfursov@qaauto01 webdriverio-browserstack2]$ sudo ps aux | grep rowser
sfursov 26407 3.3 0.2 1215364 57372 ? Ssl 16:56 0:00 /home/sfursov/.browserstack/BrowserStackLocal --daemon start --log-file /home/sfursov/webdriverio-browserstack2/local.log --key **** --force-proxy --local-proxy-host **** --local-proxy-port 8080 -daemonInstance
sfursov 26428 3.5 0.2 1214620 57160 ? Sl 16:56 0:00 /home/sfursov/.browserstack/BrowserStackLocal --log-file /home/sfursov/webdriverio-browserstack2/local.log --key **** --force-proxy --local-proxy-host **** --local-proxy-port 8080 -daemonInstance --local-identifier browserstack-fork-0
sfursov 26433 4.9 0.2 1214724 57924 ? Sl 16:56 0:00 /home/sfursov/.browserstack/BrowserStackLocal --log-file /home/sfursov/webdriverio-browserstack2/local.log --key **** --force-proxy --local-proxy-host **** --local-proxy-port 8080 -daemonInstance --local-identifier browserstack-fork-1
sfursov 26438 4.7 0.1 1214616 55256 ? Sl 16:56 0:00 /home/sfursov/.browserstack/BrowserStackLocal --log-file /home/sfursov/webdriverio-browserstack2/local.log --key **** --force-proxy --local-proxy-host **** --local-proxy-port 8080 -daemonInstance --local-identifier browserstack-fork-2
sfursov 26444 3.8 0.1 1214488 55392 ? Sl 16:56 0:00 /home/sfursov/.browserstack/BrowserStackLocal --log-file /home/sfursov/webdriverio-browserstack2/local.log --key **** --force-proxy --local-proxy-host **** --local-proxy-port 8080 -daemonInstance --local-identifier browserstack-fork-3
sfursov 26449 3.5 0.1 1214624 55628 ? Sl 16:56 0:00 /home/sfursov/.browserstack/BrowserStackLocal --log-file /home/sfursov/webdriverio-browserstack2/local.log --key **** --force-proxy --local-proxy-host **** --local-proxy-port 8080 -daemonInstance --local-identifier browserstack-fork-4
If I kill them - tests will run again