Exit code is 0 instead of 1 when 1 to n tests fail

I am running tests in the testingbot cloud using webdriver.io.

The problem

When tests are executed on a linux system, the wdio command line command exits with an exit code of 0 always, even if 1 to n tests fail. Like this our continuous integration system never knows, if the tests succeeded or failed, deploy or not deploy?

The problem does not exist on os x.

Environment

Linux: Ubuntu 12.04.5 LTS
OS X: OS X Yosemite Version 10.10.5
Node.js: 6.7
npm: 3.10.3
webdriver.io: 4.2.16
wdio-jasmine-framework: 0.2.6
wdio-selenium-standalone-service: 0.0.7
wdio-testingbot-service: 0.1.6
Tunnel-Version: 2.2

Details

One interesting detail is, that i get different logs on linux and os x, when the tunnelbot is shutting down.

Linux: "Shutting down tunnel"
OS X: "Closing TestingBot Tunnel"

Hi,

Thanks for the report. What's the command line command you're using that reproduces this problem? What's the configuration of your test?

This is the wdio binary that is not returning the correct exit code?

You should always see 'Closing TestingBot Tunnel' since this is what the testingbot-tunnel-launcher logs when closing the tunnel:

testingbot-tunnel-launcher/lib/tunnel-launcher.js

Line 143 in 8fe9d31

logger('Closing TestingBot Tunnel')

Thank you for your response!

The command i use is:
TB_KEY=XXX TB_SECRET=XXX wdio ./test/integration/wdio.conf.testingbot.js

Yes, i am calling the wdio binary only, and this is what gives me the wrong exit code.

Yes, i agree. The problem must be around there. On linux i get "Shutting down tunnel", on mac os x "Closing TestingBot Tunnel":

testingbot-tunnel-launcher/lib/tunnel-launcher.js

Line 190 in 8fe9d31

logger('Shutting down tunnel')

wdio.conf.testingbot.js:

const deepAssign = require('deep-assign');
const base = require('./wdio.conf');

exports.config = deepAssign(base.config, {
    baseUrl: 'http://localhost:9000',
    jasmineNodeOpts: {
        defaultTimeoutInterval: 30 * 1000,
    },
    maxInstances: 4,
    capabilities: [{
        browserName: 'chrome',
        version: 'latest',
        platform: 'MAC',
    }],
    services: [ 'testingbot' ],
    user: process.env.TB_KEY,
    key: process.env.TB_SECRET,
    tbTunnel: true,
    tbTunnelOpts: {
        tunnelVersion: '2.2',
    },
});

wdio.conf.js:

const debug = process.env.DEBUG || false;

exports.config = {
    debug: debug,
    specs: [
        './test/integration/specs/**/*.js',
    ],
    sync: true,
    logLevel: 'error',
    coloredLogs: true,
    screenshotPath: './test/integration/screenshots/',
    screenshotOnReject: false,
    baseUrl: 'http://localhost:8080',
    waitforTimeout: 10000,
    connectionRetryTimeout: 90000,
    connectionRetryCount: 3,
    framework: 'jasmine',
    reporters: [ 'spec' ],
    jasmineNodeOpts: {
        defaultTimeoutInterval: debug ? 24 * 60 * 60 * 1000 : 10 * 1000,
    },
    before: function(capabilities, specs) {
        require('babel-register')({
            ignore: /(cssModuleImportPreprocessor\.js|node_modules)/,
        });
    },
};

Hey. Any news on this?

Hi,

Sorry for the delay, we will try to reproduce this on Ubuntu

Awesome 👍

Hi, has there been any progress on this issue. We seem to be hitting the same problem. Running on CircleCI, when webdriver tests fail, we get exit code 0, and our build is marked as passing.

[chrome  windows 8.1 #0-0] Session ID: 614ba4b9-73f4-47e6-8b28-e515a3643687
[chrome  windows 8.1 #0-0] Spec: /home/ubuntu/frontend/test/account/demo.test.js
[chrome  windows 8.1 #0-0] Running: chrome on windows 8.1
[chrome  windows 8.1 #0-0]
[chrome  windows 8.1 #0-0]   account demo
[chrome  windows 8.1 #0-0]
[chrome  windows 8.1 #0-0]   account demo
[chrome  windows 8.1 #0-0]       1) should be possible to view demo
[chrome  windows 8.1 #0-0]
[chrome  windows 8.1 #0-0]
[chrome  windows 8.1 #0-0] 1 failing (53s)
[chrome  windows 8.1 #0-0]
[chrome  windows 8.1 #0-0] 1) account demo should be possible to view demo:
[chrome  windows 8.1 #0-0] Promise was rejected with the following reason: Error: An element could not be located on the page using the given search parameters ("h1=Demo").
[chrome  windows 8.1 #0-0] Error: Promise was rejected with the following reason: Error: An element could not be located on the page using the given search parameters ("h1=Demo").
[chrome  windows 8.1 #0-0]     at elements("h1=Demo") - getText.js:18:17
[chrome  windows 8.1 #0-0]     at getText("h1=Demo") - waitForText.js:37:22
[chrome  windows 8.1 #0-0]

Shutting down tunnel
Exit code is 0

This is the log from our CircleCI, I've added echo Exit code is $? to see the exit code.

circle.sh script:

  node prodServer &
  npm run e2e:wdio
  echo Exit code is $?

package.json:

"e2e:wdio": "./node_modules/.bin/wdio wdio.conf.js"

wdio.conf.js:

exports.config = {
  user: process.env.TB_KEY,
  key: process.env.TB_SECRET,
  services: ['testingbot'],
  tbTunnel: true,
  tbTunnelOpts: {
    tunnelVersion: '2.3'
  },
...

testingbot-tunnel-launcher@1.1.0 has been published which fixes this issue