Tests fail but gulp-notify doesn't catch 'error'

[nateritter]

I'm running into an issue when running gulp test for a function. Should be pretty simple, using gulp-codeception and gulp-notify. The issue is that when I run the command, codeception generates an error, but notify isn't triggered.

Come to think of it, this may be a gulp-codeception issue, but based on their docs, it should be passing through the 'error' to notify.onError(), so it seems it's not being caught properly in gulp-notify at first glance.

Ref: Pastebin 'test' task code: http://pastebin.com/MP4Rw76q

And then here's the actual results of the command being run...

vagrant@homestead:~/eg.dev$ gulp test
[23:26:11] Using gulpfile ~/eg.dev/Gulpfile.js
[23:26:11] Starting 'test'...
[23:26:11] Finished 'test' after 16 ms
[23:26:14] Codeception PHP Testing Framework v2.0.4
Powered by PHPUnit 4.2.2 by Sebastian Bergmann.

Functional Tests (1) --------------------------------------------------------------------------------
Trying to register as a new user (RegisterCept)                                                 Error
-----------------------------------------------------------------------------------------------------


Time: 2.51 seconds, Memory: 17.50Mb

There was 1 error:

---------
1) Failed to register as a new user in RegisterCept (/home/vagrant/eg.dev/tests/functional/RegisterCept.php)
Couldn't click "Next":
RuntimeException: Call to undefined method FunctionalTester::seeCurrentUrlEqual

Scenario Steps:
24. I click "Next"
23. I fill field {"name":"password_confirm"},"secret"
22. I fill field {"name":"password"},"secret"
21. I fill field {"name":"phone_2"},"555-555-2222"
20. I fill field {"name":"phone_1"},"555-555-1111"
19. I fill field {"name":"email"},"test@test.com"
18. I fill field {"name":"company"},"Testing, Inc."

#1  /home/vagrant/eg.dev/tests/functional/RegisterCept.php:39
#2  /home/vagrant/eg.dev/tests/functional/RegisterCept.php:39

FAILURES!                           
Tests: 1, Assertions: 11, Errors: 1.
[23:26:14] gulp-notify: [Testing Passed] All tests have passed!

[mikaelbr]

Hi and thanks for posting an issue!

Seeing their implementation and particularly these lines: https://github.com/mikeerickson/gulp-codeception/blob/master/index.js#L97-L100 - it looks like there are some documentation errors. Where as the documentation says use notify = false to get errors, the code says otherwise. Try setting notify to true and see if that helps.

As a side note: gulp-codeception shouldn't really be a gulp plugin as it is not following the gulp plugin guidelines (https://github.com/gulpjs/gulp/blob/master/docs/writing-a-plugin/guidelines.md). It simply doesn't need to be a gulp plugin, as it doesn't work on vinyl files, and could easily just be a command wrapper node module.

[nateritter]

Setting notify: true doesn't notify at all.
Setting notify: false skips the onError and notifies that everything passed.
Not setting notify has the same effect as notify: false.

For reference, I'm on v1.5.0 of gulp-notify, v0.4.3 of gulp-codeception, and v3.8.7 of gulp.

[nateritter]

Figured it out... kinda....

If you remove the testSuite param from the options, it all works as it should.

[mikaelbr]

Yeah. I think this is an issue with gulp-codeception. It could be fixed in gulp-codeception by instead of using es-map it should use through2 for handling transform streams, and emit errors when errors occur from the spawned process.