How to capture errors from testpoint data

[ORESoftware]

@isaacs

A failed testpoint object looks like so:

Result { ok: false, id: 2, name: 'yes' }

However, there is no "reason" or "error" saying why the testpoint resolved to false.

A tap parser instance will parse the stdout from some connected process, and create an object like so from the stdout. However, my question is, if there is an error trace associated with this test failure, how can we capture it with TAP? I'd like to pin the error trace to a particular test case.

IMO, (and you'd most likely agree) it's "semantically" different for an error to be raised in a test case (because of assertions which throw errors), then there is for an error to be raised outside of a test case. In that way, I would expect there to be some mechanism to pin errors to test cases, using TAP.

Perhaps that's just a limitation of TAP, however. Should we just write all errors, test case related and not, to stderr and call it a day?

[ORESoftware]

As well as errors, I am also looking to capture other information about tests, including how long the test took...but I guess TAP does not include the time information or the error that produced the test failure. I guess JSON would be better for including that information as opposed to the original TAP protocol.

[isaacs]

If the test has yaml diagnostics, then they'll be parsed and attached to the Result object.

$ tap-parser
1..1
not ok 1 - foo
  ---
  this: is diagnostic
  ...
[ [ 'plan', { start: 1, end: 1 } ],
  [ 'assert',
    Result { ok: false, id: 1, name: 'foo', diag: { this: 'is diagnostic' } } ],
  [ 'comment', '# failed 1 test\n' ],
  [ 'complete',
    FinalResults {
      ok: false,
      count: 1,
      pass: 0,
      fail: 1,
      bailout: false,
      todo: 0,
      skip: 0,
      plan: FinalPlan { start: 1, end: 1, skipAll: false, skipReason: '', comment: '' },
      failures: [ Result { ok: false, id: 1, name: 'foo', diag: { this: 'is diagnostic' } } ] } ] ]

[ORESoftware]

Ok thanks, so we'd have to include newline delimited yaml on the same line as each assertion. (Update, looks like the YAML comes on the lines following the assertion, not the same line, see: https://testanything.org/tap-specification.html)

[ORESoftware]

So I am looking for an example of how to include diagnostics in stdout. Something like this:

console.log('ok 1 foo # SKIP');
console.log(`# ${some yaml goes here}`);
console.log(`# ${some more yaml goes here}`);
console.log(`# ${some more more yaml goes here}`);
console.log(`# ${some more more more yaml goes here}`);

Is that correct? I doubt it. The reason I doubt it, is that it is unclear how the key values from the diag object get populated:

diag: { this: 'is diagnostic' } => where does key 'this' come from?

I did my due diligence and searched in the issues for "diagnostics", the only other matching issue was this one:

#19

[isaacs]

The TAP output should look like this:

1..1
not ok 1 - foo
  ---
  this: is diagnostic
  ...

The yaml output is indented by 2 spaces, starting with --- and ending with ....

http://www.node-tap.org/tap-format/#yaml-diagnostics

[ORESoftware]

Ok yeah, I am looking at this too
https://testanything.org/tap-version-13-specification.html

and it seems to be congruent with what you describe. So this yaml output is really supposed to be on separate lines eh?

Do you @isaacs happen to know if there is a node.js module that can produce yaml diagnostic output given a JS object? that would be a nice-to-have around here.