Assert.Multiple doesn't provide stacktrace for found failures

[jochenwezel]

Current results

The current output for my test in Visual Studio is as following and is missing the stacktrace line for the indiviual failed assertion:

The same output is shown at dotnet test, except that stack trace lines are not clickable

Requested results

Each asserition should

1. show the missing stacktrace (at positions A in screenshot) from the initial stack of the multiple (show in screenshot at position B) to the individual assertion line, something like following

    HHErgebnisrechnungenBpVsAwpTest.ValidateBunch1() Zeile 304
    HHErgebnisrechnungenBpVsAwpTest.AssertEqualErgebnis.<Multi>b__1_1() Zeile 103

2. in Visual Studio window, the stacktrace at positions A should be clickable (as at position B)
   

Current code snippets

Assert.Multiple is called as following:

Public Sub ErgebnisrechnungAWPvsBP(id_planjahr as Int64, id_bereich as Int64) ()
    Assert.Multiple(Sub()
                   ValidateBunch1()
                   ValidateBunch2()
              End Sub)
End Sub

Private Sub ValidateBunch1()
                    '...
                    Assert.AreEqual(Me.SummenTableBP(SumName, "rIstTMinus1"), System.Math.Round(Me.SummenTableAWP(SumName, "IstTMinus1"), 0), "BP vs AWP: " & SumName & " >> IstTMinus1")
                    Assert.AreEqual(Me.SummenTableBP(SumName, "rPlanT"), System.Math.Round(Me.SummenTableAWP(SumName, "PlanT"), 0), "BP vs AWP: " & SumName & " >> PlanT")
                    Assert.AreEqual(Me.SummenTableBP(SumName, "rHrT"), System.Math.Round(Me.SummenTableAWP(SumName, 
                    '...
End Sub

Private Sub ValidateBunch2()
                    '...
                    Assert.AreEqual(1, 2)
                    '...
End Sub

PackageReferences are as following:

    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.3.2" />
    <PackageReference Include="NUnit" Version="3.13.3" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.2.1" />

[stevenaw]

Thanks for reporting this @jochenwezel
I haven't had the chance to dig into this myself yet but there may be two separate issues here.

For your first point, I'm reminded of nunit/nunit#4242 . Do you feel like that one captures what you're looking for?

For your second point, I suspect that it may actually be an adapter issue.

[stevenaw]

Also, thank you for the clean repro and package version info. Can you also please confirm which version of Visual Studio you're using?

[jochenwezel]

For your first point, I'm reminded of nunit/nunit#4242 . Do you feel like that one captures what you're looking for?

The ticket of nunit/nunit#4242 describes an issue, that stack trace information is available for the multiple assertions, but with wrong line numbers.
In this (bug) tickete here, there is no stacktrace at all.

Can you also please confirm which version of Visual Studio you're using?

Microsoft Visual Studio Community 2022 (64-Bit), v17.6

Please note for point 2 of my request: the stacktrace line numbers are clickable usually. My intention to write down point 2 is just for presenting a complete "feature request description" ;-) Maybe the code for point 2 is already there and active as soon as the stacktrace (re-)appears in point 1.

[OsirisTerje]

The stacktrace is there, but there is something wrong either in the adapter code or in the way that the Test Explorer interprets the stacktrace.
I have added a repro here: https://github.com/nunit/nunit.issues/tree/main/Issue4406
and the dump file contains the following block:

<assertion result='Failed'>
         <message><![CDATA[  Expected: 4
  But was:  42
]]></message>
         <stack-trace><![CDATA[   at Issue4406.Tests.<>c__DisplayClass2_0.<Test1>b__0() in C:\repos\nunit\nunit.issues\Issue4406\UnitTest1.cs:line 29
   at NUnit.Framework.Assert.Multiple(TestDelegate testDelegate)
   at Issue4406.Tests.Test1() in C:\repos\nunit\nunit.issues\Issue4406\UnitTest1.cs:line 24
]]></stack-trace>
      </assertion>

This comes for each assert that fails, and contains the correct line numbers.
But it also outputs a stack trace in the failure section,

<failure>
      <message><![CDATA[Multiple failures or warnings in test:
  1)   Expected: 2
  But was:  42

  2)   Expected: 3
  But was:  42

  3)   Expected: 4
  But was:  42

  4)   Expected: 55
  But was:  5

]]></message>
      <stack-trace><![CDATA[   at Issue4406.Tests.Test1() in C:\repos\nunit\nunit.issues\Issue4406\UnitTest1.cs:line 24
]]></stack-trace>
   </failure>

It can be that this is what is being sent to the Test Explorer.

So this looks like an adapter bug, and I'll move it there.

[jochenwezel]

Partially fixed after latest update

Sorry that I haven't done it before: I recently upgraded dependencies today to

    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.6.3" />
    <PackageReference Include="NUnit" Version="3.13.3" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.5.0" />
    <PackageReference Include="NUnit.Analyzers" Version="3.6.1">

and I've got stack traces for each of the several assertions which failed :-) :

New situation

1. Stack trace is present for each failing assertion
2. Code line seems to be correct
3. BUT: the stack trace is not clickable in Visual Studio at the relevant position (see e.g. failure 1 with shown stacktrace to HHErgebnisrechnungenBpVsAwpTest.vb:line 380 instead of providing a clickable link)

[OsirisTerje]

That is pretty interesting, because I don't ;-) Hmm.. until right now, I did get a stack trace too.... this is weird.
Can you check the repro at https://github.com/nunit/nunit.issues/tree/main/Issue4406 and see if that also works for you?

But I think the reason for 3) is the way we send the stack trace and how Test Explorer expects it to be. I'll see if I can change them to be what I assume TE expects, a single stack trace line.

[jochenwezel]

Can you check the repro at https://github.com/nunit/nunit.issues/tree/main/Issue4406 and see if that also works for you?

There are at least 2 issues with the project in Issue4406:

1. nuget.config contains <add key="Local" value="c:\nuget" /> which doesn't exist (ok, solved simply by dropping this line)
2. NU1102 Package "NUnit3TestAdapter" with version (>= 4.6.0-alpha.3) not found
   • enabling beta releases in nuget still doesn't show up with this alpha asssembly
   • maybe you have to put and reference this assembly as part of this project in a subfolder of Issue4406 to allow full test of this project?

[OsirisTerje]

@jochenwezel Sorry about being to quick pushing up there. Just set the version number to 4.5.0, I didn't do any code changes, so that should work.

[Update] Just pushed up the changes, so just pull it down and it should work.

[jochenwezel]

These are the test results

SingleTest

Test1

[OsirisTerje]

Thanks! This is exactly the same as I see here. That's good.
The correct line numbers are in the 2nd line of each stacktrace. Test Explorer picks up the first line, so the way to fix this is to reduce the stacktrace to a single line identical to the 2nd line.