headless mode exits as success even when there are failures

[wawjr3d]

when there are failures in my test suite, the console reporter reports the failures correctly to the console, however the exit code for the process is success.

I traced the issue to a potential problem at https://github.com/jasmine/gulp-jasmine-browser/blob/master/src/lib/console_boot.js#L21 where this line throws an error when running headless. When you remove the line though, the process doesn't finish right away - it hangs until I manually ctrl+c out of it.

i can try digging more to make a PR but I was hoping you knew where the bug was off the top of your head

[rdy]

which headless browser were you using and what version? the results are gathered into a json object that is sent back to the gulp task by the headless browser. Normally the object can be stringified safely but its possible if there is some fatal error due to the version of the headless browser that nothing gets sent back.

i'd recommend trying to install the phantomjs npm package if you aren't already using that with gulp jasmine browser.

[wawjr3d]

I didnt specify the browser. I'm using globally installed phantomjs@1.9.18 on my local. I also see the issue in my builds on Travis: https://travis-ci.org/wawjr3d/brisket/jobs/92969065. The phantomjs that that build machine is using on travis is:

PhantomJS version
1.9.8

maybe the error is what happens at https://github.com/jasmine/gulp-jasmine-browser/blob/master/src/lib/console_boot.js#L21 ?

[wawjr3d]

@rdy i was able to fix temporarily with gulp plumber and forcing a process exit: https://github.com/bloomberg/brisket/pull/228/files. maybe adding a process exit to https://github.com/jasmine/gulp-jasmine-browser/blob/master/src/lib/drivers/phantomjs.js#L14 will solve this issue?

[rdy]

We used to have a process exit originally, but this is a bad idea especially if you have other gulp tasks that run after your jasmine one. If you exit, then other gulp tasks that follow jasmine will never be run. It is better to determine why the jasmine run is exiting in a non-standard way. This is usually due to some incompatible syntax or unexpected error. Would it be possible for you to isolate the problem in a test case I can examine? That would be helpful for me to see the exact issue.

[wawjr3d]

i found the problem! i'm doing:

var gulp = require("gulp");
var jasmineBrowser = require("gulp-jasmine-browser");
var gulpif = require("gulp-if");

function clientSideJasmine(config) {
    var specs = config.specs;
    var debug = config.debug === true;
    var src = config.src || [];
    var vendor = config.vendor || [];
    var files = [].concat(
        src,
        vendor,
        specs
    );

    return gulp.src(files)
        .pipe(jasmineBrowser.specRunner({
            console: !debug
        }))
        .pipe(gulpif(!debug, jasmineBrowser.headless()))
        .pipe(gulpif(debug, jasmineBrowser.server()));
}

module.exports = clientSideJasmine;

looks like the problem is that the last pipe always goes to gulpif regardless of the outcome of the tests and gulpif won't propagate the error. swapping the order of the 2 gulpif lines fixes the problem since the headless pipe is correctly at the end of the chain. now i get:

387 specs, 1 failure
Finished in 3.758 seconds
[14:35:11] 'test-on-client' errored after 5.84 s
[14:35:11] Error: Tests failed
    at formatError (/Users/wayne/projects/personal_work/brisket/node_modules/gulp/bin/gulp.js:169:10)
    at Gulp.<anonymous> (/Users/wayne/projects/personal_work/brisket/node_modules/gulp/bin/gulp.js:195:15)
    at emitOne (events.js:82:20)
    at Gulp.emit (events.js:169:7)
    at Gulp.Orchestrator._emitTaskDone (/Users/wayne/projects/personal_work/brisket/node_modules/gulp/node_modules/orchestrator/index.js:264:8)
    at /Users/wayne/projects/personal_work/brisket/node_modules/gulp/node_modules/orchestrator/index.js:275:23
    at finish (/Users/wayne/projects/personal_work/brisket/node_modules/gulp/node_modules/orchestrator/lib/runTask.js:21:8)
    at Stream.<anonymous> (/Users/wayne/projects/personal_work/brisket/node_modules/gulp/node_modules/orchestrator/lib/runTask.js:52:4)
    at Stream.f (/Users/wayne/projects/personal_work/brisket/node_modules/gulp/node_modules/orchestrator/node_modules/end-of-stream/node_modules/once/once.js:17:25)
    at emitOne (events.js:82:20)
    at Stream.emit (events.js:169:7)
    at Stream.reemit (/Users/wayne/projects/personal_work/brisket/node_modules/gulp-jasmine-browser/node_modules/lazypipe/node_modules/stream-combiner/node_modules/duplexer/index.js:85:16)
    at emitOne (events.js:82:20)
    at Stream.emit (events.js:169:7)
    at Stream.callee$2$0$ (/Users/wayne/projects/personal_work/brisket/node_modules/gulp-jasmine-browser/lib/headless.js:117:18)
    at tryCatch (/Users/wayne/projects/personal_work/brisket/node_modules/gulp-jasmine-browser/node_modules/babel-runtime/regenerator/runtime.js:72:40)
[14:35:11] 'default' errored after 10 s
[14:35:11] Error in plugin 'run-sequence'
Message:
    An error occured in task 'test-on-client'.
npm ERR! Test failed.  See above for more details.

Waynes:brisket wayne$ echo $?
1

[rdy]

Ah yes, I've seen this problem. I would recommend you switch to using gulp.cond instead of gulp if. The problem with gulp.if is that it executes the second function regardless of whether or not you conditionally want to apply it to the stream. Gulp cond is much better, you provided an anonymous function, it only executes that function and modifies the stream if the condition is truthy. If you switched to that I believe you wouldn't have a problem anymore. Gulp if works weirdly IMHO.

[wawjr3d]

thanks for the tip, i'll give gulp-cons a try