browserstack/browserstack-runner

Using QUnit.todo() causes build to fail instead of succeed

Krinkle opened this issue · 0 comments

Expected Behavior

Passing test.

Actual Behavior

[Windows 7, Firefox 45.0] Error: "my todo test" failed
Expected: true
  Actual: false

Details

This is happening because when browserstack-runner is consuming the testEnd event from js-reporters, it is ignoring the status field. Instead of only priting formatted errors for query.test.status === "failed", it is instead iterating over query.test.errors and always print each one as a formatted error, even if it is an expected error (such as the case with a "todo" test).

Here is the test suite for the linked example in the browser visually (passing):
capture

And here is what browserstack-runner report for that same build (failing):

https://github.com/qunitjs/qunit/runs/2753071016

[OS X Big Sur, Safari 14.0] Error: "test.todo.each [0]" failed
Expected: true
  Actual: false
  Source: runTest@http://localhost:8899/qunit/qunit.js:2262:41
	  run@http://localhost:8899/qunit/qunit.js:2250:16
	  http://localhost:8899/qunit/qunit.js:2472:19
	  processTaskQueue@http://localhost:8899/qunit/qunit.js:1860:37
	  http://localhost:8899/qunit/qunit.js:1864:29
	  promiseReactionJob@[native code]