busterjs/buster

buster-test -t for hybrid tests

dkl-ppi opened this issue · 4 comments

dkl-ppi commented

Using that hybrid configuration leads to the following weird behavior on at least linux and windows:

$ buster-test -t some-tests.js 
1 test, 1 assertion, 1 runtime ... OK
0 tests, 0 assertions, 1 runtime ... OK
$ buster-test -t some-more-tests.js 
$

Changing the order of the configuration groups in the config file leads to the following weird bevaviour:

$ buster-test -t some-more-tests.js
1 test, 1 assertion, 1 runtime ... OK // just hangs here
dkl-ppi commented

There seems to be 2 problems.

  1. A summary is written for a browser test run, even if there were no matching test files. This explains the behaviour of the first test run. I haven't investigated that problem yet.
  2. Test runs for node are not always ended correctly. The cause of that problem is, that the callback function, used in test-cli.js#runConfigGroups to iterate over the config groups, isn't called in any case. This explains the second and third test run. In both cases there is no matching test file for the node environment and as a result the end callback passed to auto-run in node.js#runTests is never called, which in turn would call the callback to process the next configuration group. I don't think we have that problem for the browser environment, but i am not entirely sure.

First attempt to solve the second problem was to filter groups without matching test files out of the config at the load:tests event, but that seems to be to late, because we are already iterating over the groups at that time. Further, no matching test files isn't the only problem. You can also have one matching test file, but without a test case declared. This leads to the same behaviour.

Instead, the problem with no matching test files for the node environment could be solved by changing the first line of node.js#runTests from

if (this.aborted) { return; }

to

if (this.aborted || rs.loadPath.paths().length === 0) { return this.done(); }

But this doesn't solve the problem with a matching test file without a test case. For that issue we have to change test-runner.js#runSuite i guess, to check the number of contexts and handle the case of no contexts specifically.

@cjohansen: Could you please have a closer look at the whole issue and tell me, what you think about it?

cjohansen commented

Re: 0 tests, I agree - we need to handle this specifically. These runners will be rewritten by @magnars and me very soon, so I really want to postpone this issue for that. There are multiple problems with the current implementations, and there's too little in common for the two runners to solve them in a meaningful way.

dkl-ppi commented

Ok, then i fix all except the case that you have a test file without a test case declared and close this issue. For the remaining case i will open a new issue and assign it to @magnars.