coyote test failure: Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.TestFramework, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

Question

coyote test failure: Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.TestFramework, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

hroe opened this issue 2 years ago · 5 comments

Hi Coyote team,
this is my first attempt to use Coyote for my project.
I read the documentation and demo's and have high hopes that Coyote will help discover bugs in our multi threaded application.
I re-implemented one of our existing unit tests to comply with Coyote requirements explained in the documentation.
E.g. create a new static method which combines the logic from the original [TestInitialize] and the [TestMethod]..

My unit tests use Microsoft.VisualStudio.TestTools.UnitTesting
From a developer command prompt, my unit tests all pass.
"C:\Build\feat124\Release_x64\proglibs>VSTest.Console.exe UnitTests\ao.CSharp2.tests.dll"
The new "coyote" unit test also succeeds as a normal unit test, provided I make it non-static.

I run coyote rewrite on all dlls in my UnitTests folder and store them in a new UnitTestsCoyote folder.
Coyote reports that some dll's are not rewritten as they are in the ignore list, such as:
Microsoft.VisualStudio.TestPlatform.TestFramework
I checked and these dll's are not modified. I assume this is expected behavior.

My rewritten unit tests still pass when running from the developer command prompt:
"C:\Build\feat124\Release_x64\proglibs>VSTest.Console.exe UnitTestsCoyote\ao.CSharp2.tests.dll"

But Coyote test fails:
`C:\Build\feat124\Release_x64\proglibs\UnitTestsCoyote>coyote test ao.CSharp2.tests.dll
Microsoft (R) Coyote version 1.5.9.0 for .NET 6.0.7
Copyright (C) Microsoft Corporation. All rights reserved.

. Testing C:\Build\feat124\Release_x64\proglibs\UnitTestsCoyote\ao.CSharp2.tests.dll.
..... Anonymized telemetry is enabled, see https://aka.ms/coyote-telemetry.
... Setting up the 'Oce.Proglibs.AO.Tests.ActiveObjectContextSwitchTestsStatic.SchedulesShouldBeExecutedInOrderStatic' test:
..... Using the random[seed:1534062935] exploration strategy.
... Running test iterations:
..... Iteration #1
..... Iteration #1 found bug #1
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.TestFramework, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
File name: 'Microsoft.VisualStudio.TestPlatform.TestFramework, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
at Oce.Proglibs.AO.Tests.ActiveObjectContextSwitchTestsStatic.SchedulesShouldBeExecutedInOrderStatic()
at Microsoft.Coyote.Runtime.CoyoteRuntime.<>c__DisplayClass59_0.b__0()
... Emitting trace-related reports:
..... Writing C:\Build\feat124\Release_x64\proglibs\UnitTestsCoyote\Output\ao.CSharp2.tests.dll\CoyoteOutput\ao.CSharp2.tests_0.txt.
..... Writing C:\Build\feat124\Release_x64\proglibs\UnitTestsCoyote\Output\ao.CSharp2.tests.dll\CoyoteOutput\ao.CSharp2.tests_0.schedule.
... Emitting coverage reports:
..... No coverage reports available.
... Testing statistics:
..... Found 1 bug.
... Scheduling statistics:
..... Explored 1 schedule: 1 fair and 0 unfair.
..... Found 100.00% buggy schedules.
..... Controlled 1 operation: 1 (min), 1 (avg), 1 (max).
..... Number of scheduling decisions in fair terminating schedules: 0 (min), 0 (avg), 0 (max).
... Elapsed 0.1643081 sec.`

I do not understand why this fails, as powershell reports that the expected version is in the UnitTestsCoyote folder:
PS C:\build\feat124\Release_x64\proglibs\UnitTestsCoyote> [System.Reflection.AssemblyName]::GetAssemblyName((Get-Item "Microsoft.VisualStudio.TestPlatform.TestFramework.dll").FullName).FullName Microsoft.VisualStudio.TestPlatform.TestFramework, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a

Can you give guidance on how to solve this problem?

Thanks a lot, Herman

Answer 1 · 2022-08-15T23:49:11.000Z

Hi @hroe, thanks for your interest in using Coyote & lettting us know about this issue!

There are two ways to write a test with Coyote: (1) use the Microsoft.Coyote.SystematicTesting.Test attribute from the Coyote namespace to declare your test and then use the coyote test tool to invoke this test (this is what you are doing above) and (2) use the Microsoft.Coyote.SystematicTesting.TestingEngine API to declare your test and then run it from some other unit testing framework (like MSTest/VSTest, xUnit, nUnit, etc), as if it was any other unit test.

You can read more about this 2nd approach in our docs here.

There are various benefits with this 2nd approach, including that it can integrate with existing unit testing frameworks & existing CI pipelines. Another benefit is that it can be used to resolve such assembly loading issues like the one that you are facing. I think this issue might be caused because you are trying to load the Microsoft.VisualStudio.TestPlatform.TestFramework.dll (which is for MSTest/VSTest), which would be more appropriate when goind down the 2nd route. Could you consider using the TestingEngine and let us know if this resolved your issue?

Note that going down the 2nd route, you are still expected to run coyote rewrite as a post-build task. (And you are correct that the Microsoft.VisualStudio.TestPlatform.TestFramework DLL not being modified via coyote rewrite is expected.)

Answer 2 · 2022-08-16T09:44:51.000Z

Hi @pdeligia, thanks for explaining the two main approaches.
Meanwhile I stripped all dependencies on MSTest/VSTest and was able to run it using coyote test (e.g. option 1).

I will also look into option 2) as it seems more flexible and might allow me to reuse more logic from our existing tests.
I will keep you updated.

In general, should I ask guidance by opening specifc issues like I did for this one, or do you prefer that I create a new discussion topic for each question? For example, I want to elaborate on the best way to add support for ReaderWriterLockSlim by better understanding the approach implemented in SemaphoreSlim.cs.

Answer 3 · 2022-08-16T15:57:40.000Z

Sounds great @hroe, glad it was helpful.

And sounds good, either opening a new issue or a discussion topic is completely fine with us. Saying this, we have not really used the discussions feature much before, but it seems more fitting for what you are saying here regarding ReaderWriterLockSlim support.

Answer 4 · 2022-08-24T11:58:20.000Z

@pdeligia, option 2, unit testing approach worked well and solved the problem. Below an example [TestMethod].

   [TestMethod]
    public void TestPhase5InitStorageGoodConfiguration()
    {
        var config = Microsoft.Coyote.Configuration.Create()
            .WithTraceVisualizationEnabled()
            .WithTestIterationsRunToCompletion()
            //.WithRandomGeneratorSeed(<<seed_to_replay_a_failure>>)
            .WithTestingIterations(10);
        var engine = Microsoft.Coyote.SystematicTesting.TestingEngine.Create(config, TestPhase5InitStorageGoodConfigurationCore);
        engine.Run();
        var report = engine.TestReport;
        IEnumerable<string>reportPaths;
        engine.TryEmitReports("coyoteReports", "Orchestrator", out reportPaths);
        Console.WriteLine("Coyote found {0} bug, #reports {1}", report.NumOfFoundBugs, reportPaths.Count());
        foreach (var r in reportPaths)
        {
            Console.WriteLine(r);
        }
        Assert.AreEqual(0, report.NumOfFoundBugs);
    }

I use config.WithTraceVisualizationEnabled() to so can analyse using DGML graphs in case of a failure. But although the test fails, reportsPaths.Count() remains zero. How can I get access to the DGML graphs?

I think I have to recompile the test case with config.WithRandomGeneratorSeed(<<seed_to_replay_a_failure>>) to debug a failure. Is there a way to use config settings file for the test? Then I can re-run and debug the failure without re-compilation.

Answer 5 · 2022-08-25T22:29:43.000Z

Glad it worked @hroe!

Regarding WithTraceVisualizationEnabled, sadly this DGML feature is only available for the optional Actor library that is provided as part of Coyote, and cannot be used with regular C# tasks. This feature requires a graph logger to run during systematic testing and that is only implemented today for actors & state machines (capturing their transitions). We would like to support this at some point in the future, but there is no ETA for this I am afraid due to the inherent complexity of capturing a task-based program in a DGML form and what that would even look like (but we also welcome community contributions, if someone wants to attempt this!).

Actually, to debug a failure, all you need is the "reproducible trace". You can also do this with the seed (as you say), but this only works for the default random exploration strategy. The trace is the most reliable way to repro a bug. See this part here in the docs. To get a trace from a failed execution, you can grab the engine.ReproducibleTrace property and then feed it to Configuration.Create().WithReplayStrategy(trace) (from the above link) next time you run the test via the test engine. You can also dump this trace to a file with engine.TryEmitReports(...) to debug at a later point (or from different machine, e.g. if something failed on CI).

Coyote does not really provide any extra infrastructure/helpers on top of the TestingEngine besides these above APIs for bug replay, but this is something that you could implement on top in your code? E.g., you could write your test to detect if this replay trace (file) exists (perhaps through an environment variable? but there could be other ways too) and follow the replay path?