Child Parsers emit complete before assertions are emitted
Closed this issue · 1 comments
Hi all,
I am working on a project where grouping results by subtest is preferred. I am finding that it is quite difficult to track and group assertions by subtests starting and ending.
The parser will correctly emit a child parser, but the child parser itself does not emit assertions / pass / fails that it finds. It also emits its "complete" event before the assertions are emitted from the top level parser.
For example given this example parser:
#!/usr/bin/env node
const Parser = require('tap-parser');
const through = require('through2');
const duplexer = require('duplexer');
const Formatter = () => {
const p = new Parser();
const out = through.obj();
const dup = duplexer(p, out);
const listenToChild = (c) => {
let test;
let depth;
return () => {
c.on('comment', (comment) => {
if (/[\s]*\#\sSubtest:\s(.+)/.test(comment)) {
//console.log('TEST', RegExp.$1);
test = RegExp.$1;
console.log('START', test);
}
});
c.on('child', (childParser) => listenToChild(childParser)());
c.on('complete', () => {
console.log('CHILD END', test);
});
}
}
p.on('child', (childParser) => listenToChild(childParser)());
p.on('pass', (assertion) => {
console.log('pass', assertion);
});
p.on('fail', (assertion) => {
console.log('fail', assertion);
});
return dup;
};
process.stdin.pipe(Formatter());
The assertions appear AFTER the child emits an end. This makes it quite difficult to write an algo that will catch any assertions (pass or fail) within a known buffer.
I am trying to create some base parser logic that will start a recursive stack on the beginning of a subtest, then any assertions get thrown in an array / map for the specific test depth we are on. Once it detects that the current child has ended it will unwind the stack and build a data structure that has the assertions grouped by subtest.
I have looked over all of the example Tap parsers and none of them group by test, they assume the result is a stream of assertions at the top level. Which makes sense, as the test that are reporting in the console are linear, and read line for line like the tap output. But this limitation is potentially stopping parsers from being able to group results by subtests.
I hope that makes sense... :/
Here is an example:
Test:
const tap = require('tap').mochaGlobals();
const should = require('should');
const assumeThisWasTheAsyncData = ['subject1', 'subject2', 'subject3'];
describe('TOP LEVEL', () => {
assumeThisWasTheAsyncData.forEach((subject) => {
describe(`GROUPED ${subject}`, () => {
if (true) { // SIMPLIFIED
context('CONTEXT WHEN INITIAL CONDITIONAL IS TRUE', () => {
it(`A TEST for ${subject}`, () => {
should.equal(true, true, `assertion should be true`);
});
});
} else {
it(`A TEST for ${subject}`, () => {
should.ok(true, 'assertion is true as we skipped conditional');
});
}
});
});
});
In my actual test describe is async and it awaits assumeThisWasTheAsyncData, this is an interesting and amazing feature of tap as its so low level it can allow async describe blocks with tests inside, unlike say Jest or Jasmine. See jasmine/jasmine#1487.
Parser Output:
START ./test/simple-bdd.test.js
START TOP LEVEL
START GROUPED subject1
START CONTEXT WHEN INITIAL CONDITIONAL IS TRUE
START A TEST for subject1
CHILD END A TEST for subject1
pass Result {
ok: true,
id: 1,
time: 0.821,
buffered: true,
name: 'A TEST for subject1',
fullname:
'./test/simple-bdd.test.js TOP LEVEL GROUPED subject1 CONTEXT WHEN INITIAL CONDITIONAL IS TRUE' }
CHILD END CONTEXT WHEN INITIAL CONDITIONAL IS TRUE
CHILD END GROUPED subject1
START GROUPED subject2
START CONTEXT WHEN INITIAL CONDITIONAL IS TRUE
START A TEST for subject2
CHILD END A TEST for subject2
pass Result {
ok: true,
id: 1,
time: 0.058,
buffered: true,
name: 'A TEST for subject2',
fullname:
'./test/simple-bdd.test.js TOP LEVEL GROUPED subject2 CONTEXT WHEN INITIAL CONDITIONAL IS TRUE' }
CHILD END CONTEXT WHEN INITIAL CONDITIONAL IS TRUE
CHILD END GROUPED subject2
START GROUPED subject3
START CONTEXT WHEN INITIAL CONDITIONAL IS TRUE
START A TEST for subject3
CHILD END A TEST for subject3
pass Result {
ok: true,
id: 1,
time: 0.042,
buffered: true,
name: 'A TEST for subject3',
fullname:
'./test/simple-bdd.test.js TOP LEVEL GROUPED subject3 CONTEXT WHEN INITIAL CONDITIONAL IS TRUE' }
CHILD END CONTEXT WHEN INITIAL CONDITIONAL IS TRUE
CHILD END GROUPED subject3
CHILD END TOP LEVEL
CHILD END ./test/simple-bdd.test.js
Tap Output:
TAP version 13
ok 1 - ./test/simple-bdd.test.js # time=587.364ms {
ok 1 - TOP LEVEL # time=33.498ms {
ok 1 - GROUPED subject1 # time=9.966ms {
ok 1 - CONTEXT WHEN INITIAL CONDITIONAL IS TRUE # time=6.642ms {
ok 1 - A TEST for subject1 # time=0.821ms {
1..0
}
1..1
}
1..1
}
ok 2 - GROUPED subject2 # time=6.689ms {
ok 1 - CONTEXT WHEN INITIAL CONDITIONAL IS TRUE # time=3.506ms {
ok 1 - A TEST for subject2 # time=0.058ms {
1..0
}
1..1
}
1..1
}
ok 3 - GROUPED subject3 # time=6.083ms {
ok 1 - CONTEXT WHEN INITIAL CONDITIONAL IS TRUE # time=3.056ms {
ok 1 - A TEST for subject3 # time=0.042ms {
1..0
}
1..1
}
1..1
}
1..3
}
1..1
# time=37.706ms
}
1..1
# time=605.075ms
----------|----------|----------|----------|----------|-------------------|
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s |
----------|----------|----------|----------|----------|-------------------|
All files | 0 | 0 | 0 | 0 | |
----------|----------|----------|----------|----------|-------------------|
Let me know if you need anymore context. I hope I explained it well enough.
I think you probably want to be using the complete event rather than the stream end event. The end event is simply signifying that all of the input has been consumed and thus no more should be written, but the processing of that input happens after it's been consumed.
Something like this: https://gist.github.com/isaacs/e4cd8a418372e5a15235e33347de8d06
We could defer end events until after the parser emits complete, but it'd be kind of complicated, and I'm not sure it provides much value.