Dependencies not loaded with dotnet cli runner
slorion opened this issue · 21 comments
Using dotnet nbench <...>, dependencies are not loaded correctly, both nuget packages and project references. It works if all dll are added manually in the right order on the command line.
@slorion I can reproduce using NuGet Install-Package NBench.Runner. However, for the .NET Core version, I am able to successfully reference and use it with the correct <DotNetCliToolReference> in .csproj:
<ItemGroup>
<DotNetCliToolReference Include="NBench.Runner.DotNetCli" Version="1.0.0" />
</ItemGroup>
We will troubleshoot the NuGet version issue ASAP. Thanks.
FYI, I manually added <DotNetCliToolReference> inside the csproj and then did dotnet restore. I did not use Install-Package.
No problem, no rush needed, we are experimenting with this and https://github.com/Microsoft/xunit-performance
@slorion Are you running dotnet nbench from within the project's parent folder? This works in my testing.
When doing so, I got an error message saying that dotnet-nbench could not be found. I will retry soon and let you know for sure.
Ok, syntax should be dotnet nbench with a space, just like dotnet build, run, etc
Yes, this is the syntax I used, but the error message did contain a dash between dotnet and nbench.
Using VS2017 with .NET Core 1.1.1 (SDK 1.0.1)
@slorion I've tested a few times now, with a cleared NuGet cache and I'm still able to run dotnet nbench in the client project. Can you verify that you are opening a shell inside of the project folder and attempting to run the tool? The tool will not be installed on your PATH, so it must be run inside the .NET Core project that pulled in the package
Just retested it.
Inside the folder with the csproj, it works fine. In the parent of that folder (I guess this is what you meant by "project's parent folder"), I get No executable found matching command "dotnet-nbench".
And then running dotnet nbench .\bin\debug\netcoreapp1.1\<my dll> from the folder with the csproj, I get an exception System.IO.FileNotFoundException: Could not load file or assembly <...>
@slorion Ok thanks for testing. Yes, I meant the .csproj folder of the project that uses the tool. This is expected behavior, so I will update the documentation to make it more clear. Thanks again.
Ok thank you
Is dotnet nbench also fixed ? In the readme.md, it still indicates to use
<ItemGroup>
<DotNetCliToolReference Include="NBench.Runner.DotNetCli" Version="1.0.0" />
</ItemGroup>
I tried with both 1.0.0 and 1.0.1 and without changing anything else, the error I now have is
ERROR: Error while executing the tests.
System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
at System.Reflection.RuntimeAssembly.get_DefinedTypes()
at NBench.Sdk.Compiler.ReflectionDiscovery.ClassesWithPerformanceBenchmarks(Assembly targetAssembly)
at NBench.Sdk.Compiler.ReflectionDiscovery.FindBenchmarks(Assembly targetAssembly)
at NBench.Sdk.TestRunner.Execute()
@slorion dotnet nbench wasn't affected by the patch since it was working as intended. CI tests show that it is operational also so I will probably need more info about how you're using the netcoreapp1.0 version. i.e. what type of test assembly are you running it against, is that assembly compiled as a netcoreapp1.x, etc. Based on the error message, it looks like the dotnet nbench runner is working, but it's having issues loading the benchmark assembly. Thanks.
@heynickc , I don't know what to say more than I am using it with a standard class library project created with VS 2017, targeting .NET Core 1.1.
I added the CLI tool reference as indicated and the tested assembly contains a single test with exactly the same attributes as in the example in the readme file. I add the CLI reference, call dotnet restore in the project folder and then run dotnet nbench .\bin\debug\netcoreapps1.1\<my dll>.
You mention
I will probably need more info about how you're using the netcoreapp1.0 version
I am in fact using .NET Core 1.1 with SDK 1.0.1. Maybe that is the source of the difference in behavior ?
@slorion Yes, the dotnet nbench runner currently supports netcoreapp1.0 targets. I haven't tested later versions. Can you drop your benchmark assembly back to netcoreapp1.0 and test this when you get the chance? Thanks.
I trust you it does run on .NET Core 1.0. We are using .NET Core 1.1 and the new csproj format, so moving back to 1.0 would not help us much. I know there has been a lot of back and forth from Microsoft with .NET Core on many aspects, so I understand perfectly the incompatibility issues.