getgauge/gauge-csharp

Add linux mono support

Closed this issue · 20 comments

tomzo commented

It would be nice if running on mono was supported. I have initially attempted to do so and found several problems:

  • there is no build scripts for linux at the moment, but solution can be easily build with monodevelop or xbuild.
  • zip package is packaged with windows path separators \. So when installed with gauge --install csharp --file .. it results in bad paths on linux, e.g. a file bin\Gauge.CSharp.Runner.exe instead of directory bin and bin/Gauge.CSharp.Runner.exe file
  • after fixing above installed paths. There plugin config ,must be updated to something like this ~/.gauge/plugins/csharp/0.7.2/csharp.json. Unfortunately only full paths work
"run": {
    "windows": ["bin\\Gauge.Csharp.Runner.exe", "--start"],
    "linux": ["/usr/bin/mono", "<full-path-to-home>.gauge/plugins/csharp/0.7.2/bin/Gauge.CSharp.Runner.exe", "--start"],
    "darwin": ["echo", "Coming Soon!"]
  },

And with all above changes, the runner still fails. I tried executing from IntegrationTestSample available in this project.

gauge-csharp/IntegrationTestSample$ gauge --verbose=true specs/
Gauge.CSharp.Runner.GaugeProjectBuilder Building Project: /ide/work/gauge-csharp/IntegrationTestSample/IntegrationTestSample.csproj
/ide/work/gauge-csharp/IntegrationTestSample/IntegrationTestSample.csproj: error : Unhandled exception occured during a build
Gauge.CSharp.Runner.GaugeProjectBuilder Failure

I haven figured out yet what is failing when project build is attempted. It can be compiled in monodevelop.

Question to maintainers and contributors - Is there something else that would be needed for linux/mono support, that you know of?
It seems none of above problems is critical, so I could make a PR once I get this to work.

sriv commented

@tomzo - nice to see this get started.

I have played around briefly, but never got around to make this production ready.

On your points:

there is no build scripts for linux at the moment, but solution can be easily build with monodevelop or xbuild.

I believe the problem you are facing is because of Gauge-CSharp using Microsoft.Build.Framework APIs. MSBuild does not work on *nix, and need to see an alternative that is truly cross platform.

The relevant code is in GaugeProjectBuilder

zip package is packaged with windows path separators . So when installed with gauge --install csharp --file .. it results in bad paths on linux, e.g. a file bin\Gauge.CSharp.Runner.exe instead of directory bin and bin/Gauge.CSharp.Runner.exe file

Yep, everything in our build script needs to be made cross platform.

after fixing above installed paths. There plugin config ,must be updated to something like this ~/.gauge/plugins/csharp/0.7.2/csharp.json. Unfortunately only full paths work

Strange, I have mono in my PATH (in OSx). Here is an extract from my csharp.json:

  "run": {
    "windows": ["bin\\Gauge.Csharp.Runner.exe", "--start"],
    "linux": ["echo", "Coming Soon!"],
    "darwin": ["/usr/local/bin/mono", "bin/Gauge.Csharp.Runner.exe", "--start"]
  },
  "init": {
    "windows": ["bin\\Gauge.Csharp.Runner.exe", "--init"],
    "linux": ["echo", "Coming Soon!"],
    "darwin": ["/usr/local/bin/mono", "bin/Gauge.Csharp.Runner.exe", "--init"]
  },

I could get a basic execution working by following the below steps:

  • cd /tmp/bar
  • gauge --init csharp
  • xbuild bar.csproj
  • gauge_custom_build_path=bin/Debug gauge specs

Init Output

INb:bar steam$ gauge --init csharp
 create  bar
 create  Properties/AssemblyInfo.cs
 create  StepImplementation.cs
 create  packages.config
 create  bar.csproj
 create  bar.sln
 Installing Nuget Package : FluentAssertions, version: 3.5.0
 Installing Nuget Package : Gauge.CSharp.Lib, version: 0.5.3
 Done Installing Nuget Package!
 create  manifest.json
 create  specs
 create  specs/example.spec
 create  env
 create  env/default
 create  env/default/default.properties
Successfully initialized the project. Run specifications with "gauge specs/"

Build Output

XBuild Engine Version 12.0
Mono, Version 3.12.0.0
Copyright (C) 2005-2013 Various Mono authors

Build started 27-05-2016 13:06:15.
__________________________________________________
Project "/private/tmp/bar/bar.csproj" (default target(s)):
    Target PrepareForBuild:
        Configuration: Debug Platform: AnyCPU
        Created directory "bin/Debug/"
        Created directory "obj/Debug/"
    Target CopyFilesMarkedCopyLocal:
        Copying file from '/private/tmp/bar/packages/FluentAssertions.3.5.0/lib/net45/FluentAssertions.dll' to '/private/tmp/bar/bin/Debug/FluentAssertions.dll'
        Copying file from '/private/tmp/bar/packages/FluentAssertions.3.5.0/lib/net45/FluentAssertions.Core.dll' to '/private/tmp/bar/bin/Debug/FluentAssertions.Core.dll'
        Copying file from '/private/tmp/bar/packages/Gauge.CSharp.Lib.0.5.3.0/lib/net45/Gauge.CSharp.Lib.dll' to '/private/tmp/bar/bin/Debug/Gauge.CSharp.Lib.dll'
    Target GenerateSatelliteAssemblies:
    No input files were specified for target GenerateSatelliteAssemblies, skipping.
    Target CoreCompile:
        Tool /usr/local/Cellar/mono/3.12.0/lib/mono/4.5/mcs.exe execution started with arguments: /noconfig /debug:full /debug+ /optimize- /out:obj/Debug/bar.dll StepImplementation.cs Properties/AssemblyInfo.cs obj/Debug/.NETFramework,Version=v4.5.AssemblyAttribute.cs /target:library /define:"DEBUG;TRACE" /nostdlib /reference:packages/FluentAssertions.3.5.0/lib/net45/FluentAssertions.dll /reference:packages/FluentAssertions.3.5.0/lib/net45/FluentAssertions.Core.dll /reference:packages/Gauge.CSharp.Lib.0.5.3.0/lib/net45/Gauge.CSharp.Lib.dll /reference:/usr/local/Cellar/mono/3.12.0/lib/mono/4.5/System.dll /reference:/usr/local/Cellar/mono/3.12.0/lib/mono/4.5/System.Xml.Linq.dll /reference:/usr/local/Cellar/mono/3.12.0/lib/mono/4.5/System.Data.DataSetExtensions.dll /reference:/usr/local/Cellar/mono/3.12.0/lib/mono/4.5/Microsoft.CSharp.dll /reference:/usr/local/Cellar/mono/3.12.0/lib/mono/4.5/System.Data.dll /reference:/usr/local/Cellar/mono/3.12.0/lib/mono/4.5/System.Xml.dll /reference:/usr/local/Cellar/mono/3.12.0/lib/mono/4.5/System.Core.dll /reference:/usr/local/Cellar/mono/3.12.0/lib/mono/4.5/mscorlib.dll /warn:4
    Target DeployOutputFiles:
        Copying file from '/private/tmp/bar/obj/Debug/bar.dll.mdb' to '/private/tmp/bar/bin/Debug/bar.dll.mdb'
        Copying file from '/private/tmp/bar/obj/Debug/bar.dll' to '/private/tmp/bar/bin/Debug/bar.dll'
Done building project "/private/tmp/bar/bar.csproj".

Build succeeded.
     0 Warning(s)
     0 Error(s)

Time Elapsed 00:00:00.7010480

gauge specs Output

INb:bar steam$ gauge_custom_build_path=bin/Debug gauge specs
Gauge.CSharp.Runner.SandboxFactory Creating a Sandbox in: /tmp/bar/bin/Debug
# Specification Heading
  ## Vowel counts in single word     ✔ ✔
  ## Vowel counts in multiple word   ✔ ✔

Successfully generated html-report to => /tmp/bar/reports/html-report
Specifications: 1 executed  1 passed    0 failed    0 skipped
Scenarios:  2 executed  2 passed    0 failed    0 skipped

Total time taken: 121ms
tomzo commented

Hi @sriv,
Thank you for your help.

I have several projects which I maintain on linux and windows. At some point I started using F# FAKE and paket for build system and I'd say there is nothing to complain. So my suggestion on solving cross-platform build problems would be to leverage those projects.

GaugeProjectBuilder

I think it could be implemented using FAKEs msbuildhelper. After all this class responsibility is to find *.csproj project file and build it.

Yep, everything in our build script needs to be made cross platform.

As I said, I think it is best to move to FAKE + paket. I can start that if you like.

Full paths

I checked again, it works. I must have done something wrong before.

I was able to execute gauge runner in IntegrationTestSample and generate report. csharp.json was

{
  "id" : "csharp",
  "version" : "0.7.2",
  "description": "C# support for gauge",
  "install": {
    "windows": [],
    "linux": [],
    "darwin": []
  },
  "run": {
    "windows": ["bin\\Gauge.Csharp.Runner.exe", "--start"],
    "linux": ["/usr/bin/mono", "bin/Gauge.CSharp.Runner.exe", "--start"],
    "darwin": ["echo", "Coming Soon!"]
  },
  "init": {
    "windows": ["bin\\Gauge.Csharp.Runner.exe", "--init"],
    "linux": ["echo", "Coming Soon!"],
    "darwin": ["echo", "Coming Soon!"]
  },
  "gaugeVersionSupport": {
      "minimum": "0.2.0",
      "maximum": ""
  }
}
sriv commented

FAKE + paket would be great, please proceed and keep us posted on how you get along with this. And thanks for signing up for this contribution, much appreciated.

tomzo commented

I'll be working on convertion to FAKE and paket today.
@sriv can you tell me more about Gauge.CSharp.Core project? Why is treated specially? Do you want it to be build together with all other projects? Does it need a nuget package?

sriv commented

@tomzo - Neat.

Regarding Gauge.CSharp.Core - it contains some classes/methods that help communicate with Gauge Core's API. Two separate components use it: a) Gauge CSharp Runner and b)Gauge VisualStudio plugin. Hence it is extracted out as a separate project and hosted in Nuget just for ease of package management.

The core package is not supposed to be used by an end user, hence it is marked as "Internal".
Does this help?

tomzo commented

Thanks. I am trying to understand what end results we want from build targets in this project.
So there should be

  • one "chain" where I build Gauge.Csharp.Core project and package it as nuget.
  • one "chain" where I build Gauge.Csharp.Lib, run tests, and package it as another nuget.
  • finaly runner, which I build, unit test, integration test and package zip with the skeleton

Is that more or less correct?

sriv commented

That sounds like a good start, we need to have targets to run Functional Tests (refer here) and Regenerate proto classes (whenever there is any change in proto contracts), but I guess those can be added later.

tomzo commented

Functional tests

OK. I don't think these will be a problem.

I have managed so far to create build and test tasks. Currently 26 tests are failing on linux due to path separators, I think these error are all because of asserts written with \ separator, so that will be easy to fix.

I am building this on linux and windows as I work. I got stuck about an hour ago with assembly loading problem on windows. Please have a look at output below. Did you have any errors like this before?

...
05:21:04.109 C:\GoAgent\pipelines\gauge-csharp\packages\test\NUnit.Runners\tools\nunit-console.exe "-nologo" "-noshadow" "-labels" "C:\GoAgent\pipelines\gauge-csharp\artifacts\gauge-csharp\bin\Gauge.CSharp.Runner.IntegrationTests.dll" "C:\GoAgent\pipelines\gauge-csharp\artifacts\gauge-csharp\bin\Gauge.CSharp.Runner.UnitTests.dll" "C:\GoAgent\pipelines\gauge-csharp\artifacts\gauge-csharp\bin\IntegrationTestSample.dll" "-xml:artifacts/gauge-csharp/bin/gauge.csharp.runner.unittests.xml" 
05:21:04.406 ProcessModel: Default    DomainUsage: Multiple
05:21:04.406 Execution Runtime: net-3.5
05:21:06.171 Unhandled Exception:
05:21:06.171 System.IO.FileLoadException: Could not load file or assembly 'Gauge.CSharp.Lib, Version=0.5.3.0, Culture=neutral, PublicKeyToken=3f92af01a0ea350e' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
05:21:06.171 File name: 'Gauge.CSharp.Lib, Version=0.5.3.0, Culture=neutral, PublicKeyToken=3f92af01a0ea350e'
05:21:06.171 
05:21:06.171 Server stack trace: 
05:21:06.171    at System.ModuleHandle.ResolveType(RuntimeModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
05:21:06.171    at System.ModuleHandle.ResolveTypeHandleInternal(RuntimeModule module, Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
05:21:06.171    at System.ModuleHandle.ResolveTypeHandle(Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
05:21:06.171    at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
05:21:06.171    at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(CustomAttributeRecord caRecord, MetadataImport scope, Assembly& lastAptcaOkAssembly, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, Object[] attributes, IList derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctor, Boolean& ctorHasParameters, Boolean& isVarArg)
05:21:06.171    at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType, Boolean mustBeInheritable, IList derivedAttributes, Boolean isDecoratedTargetSecurityTransparent)
05:21:06.171    at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeMethodInfo method, RuntimeType caType, Boolean inherit)
05:21:06.171    at NUnit.Core.Reflect.GetAttributes(ICustomAttributeProvider member, Boolean inherit)
05:21:06.171    at NUnit.Core.Reflect.HasAttribute(ICustomAttributeProvider member, String attrName, Boolean inherit)
05:21:06.171    at NUnit.Core.Reflect.GetMethodsWithAttribute(Type fixtureType, String attributeName, Boolean inherit)
05:21:06.171    at NUnit.Core.NUnitTestFixture..ctor(Type fixtureType, Object[] arguments)
05:21:06.171    at NUnit.Core.Builders.NUnitTestFixtureBuilder.BuildSingleFixture(Type type, Attribute attr)
05:21:06.171    at NUnit.Core.Builders.NUnitTestFixtureBuilder.BuildFrom(Type type)
05:21:06.171    at NUnit.Core.Extensibility.SuiteBuilderCollection.BuildFrom(Type type)
05:21:06.171    at NUnit.Core.TestFixtureBuilder.BuildFrom(Type type)
05:21:06.171    at NUnit.Core.Builders.TestAssemblyBuilder.GetFixtures(Assembly assembly, String ns)
05:21:06.171    at NUnit.Core.Builders.TestAssemblyBuilder.Build(String assemblyName, Boolean autoSuites)
05:21:06.171    at NUnit.Core.Builders.TestAssemblyBuilder.Build(String assemblyName, String testName, Boolean autoSuites)
05:21:06.171    at NUnit.Core.TestSuiteBuilder.BuildSingleAssembly(TestPackage package)
05:21:06.171    at NUnit.Core.TestSuiteBuilder.Build(TestPackage package)
05:21:06.171    at NUnit.Core.SimpleTestRunner.Load(TestPackage package)
05:21:06.171    at NUnit.Core.ProxyTestRunner.Load(TestPackage package)
05:21:06.171    at NUnit.Core.ProxyTestRunner.Load(TestPackage package)
05:21:06.171    at NUnit.Core.RemoteTestRunner.Load(TestPackage package)
05:21:06.171    at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
05:21:06.171    at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
05:21:06.171 
05:21:06.171 Exception rethrown at [0]: 
05:21:06.171    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
05:21:06.171    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
05:21:06.171    at NUnit.Core.TestRunner.Load(TestPackage package)
05:21:06.171    at NUnit.Util.TestDomain.Load(TestPackage package)
05:21:06.171    at NUnit.Util.AggregatingTestRunner.Load(TestPackage package)
05:21:06.171    at NUnit.Core.ProxyTestRunner.Load(TestPackage package)
05:21:06.171    at NUnit.Util.RemoteTestAgent.AgentRunner.Load(TestPackage package)
05:21:06.171    at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
05:21:06.171    at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
05:21:06.171 
05:21:06.171 Exception rethrown at [1]: 
05:21:06.171    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
05:21:06.171    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
05:21:06.171    at NUnit.Core.TestRunner.Load(TestPackage package)
05:21:06.171    at NUnit.Core.ProxyTestRunner.Load(TestPackage package)
05:21:06.171    at NUnit.Util.ProcessRunner.Load(TestPackage package)
05:21:06.171    at NUnit.ConsoleRunner.ConsoleUi.Execute(ConsoleOptions options)
05:21:06.171    at NUnit.ConsoleRunner.Runner.Main(String[] args)
05:21:06.171 
05:21:06.171 WRN: Assembly binding logging is turned OFF.
05:21:06.171 To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
05:21:06.171 Note: There is some performance penalty associated with assembly bind failure logging.
05:21:06.171 To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
05:21:06.171 
05:21:06.234 Running build failed.
05:21:06.234 Error:
05:21:06.234 NUnit test failed. Process finished with exit code UnexpectedError (-100).
05:21:06.249 
05:21:06.249 ---------------------------------------------------------------------
05:21:06.249 Build Time Report
05:21:06.249 ---------------------------------------------------------------------
05:21:06.249 No target was successfully completed
05:21:06.249 ---------------------------------------------------------------------
05:21:06.812 [script-executor] Script completed with exit code: 42.
tomzo commented

This also happens when starting runner from command line, so I might be able to debug.

Another question, wouldn't it be easier if there was just one solution for all projects? Was there any particualr reason to have 5 solutions?
With 1 solution, all references could be by project. (not HintPath)

tomzo commented

So the problem is that, there is a commited binary at https://github.com/getgauge/gauge-csharp/blob/master/IntegrationTestSample/Lib/Gauge.CSharp.Lib.dll which was causing conflicts with newly built version of Gauge.CSharp.Lib.dll.
The history of commits of that file does not say much, @sriv can you tell me what was the vision for lifecycle of this file? Would it be OK if I removed it from git and add a FAKE target that installs it from artifacts/ directory?

tomzo commented

I also noticed, current distribution zip archives contain all test assemblies and even the xml output from test runs, do we really want them included? I can easily take of this.

sriv commented

Hi @tomzo , some responses:

wouldn't it be easier if there was just one solution for all projects? Was there any particualr reason to have 5 solutions

The reason for various solutions is to generate artifacts without having to compile other. ex. Gauge.CSharp.Core.sln is used just to generate the Core nupkg.

I am happy to consolidate this, if you feel the newer build tasks could take care of atomic project packaging and generate the artifacts.

Committed Gauge.CSharp.Lib.dll.. Would it be OK if I removed it from git and add a FAKE target that installs it from artifacts/ directory?

So, the purpose of this checked in dll is just one - it serves as input for a test that ensures that Gauge.CSharp.Runner works when using a Lib version that is different from the Lib that the user's Gauge Csharp uses. If you notice the version of the checked-in Gauge.CSharp.Lib.dll, it would be 0.0.0. This test uses this information.

If you can think of an alternative to either mock/stub or simulate a different version of the dll, we could take this file out. Thoughts?

I also noticed, current distribution zip archives contain all test assemblies and even the xml output from test runs, do we really want them included? I can easily take of this.

Good catch, definitely not intentional. We should only package binaries required for execution.

tomzo commented

Thanks for your answers, that clarifies a few things.

Gauge.CSharp.Lib.dll

So it works a little bit like regression test. It isn't so bad being in SCM after you explained it. However I cannot get that Runner.IntegrationTests assembly to load on windows. The only time it worked, was when I replaced checked-in assembly with a new one, but I see now how that is wrong.

If you can think of an alternative to either mock/stub or simulate a different version of the dll, we could take this file out. Thoughts?

Fake has good support for generating AssemblyInfo with any version you want. So we could compile assembly twice with 2 different versions.
Another option would be to just add a strict reference in IntegrationTestSample on old already published nuget Gauge.CSharp.Lib package. E.g. we compile IntegrationTestSample with Gauge.CSharp.Lib 0.0.2 and then test that such version was loaded. Do you think that would keep the value of tests?

I am going to leave the windows integration test problem for a moment.
I'll add zip and nuget packaging targets and make a PR with FAKE-only changes.
Then I will start working on fixing linux tests and approach the GaugeProjectBuilder to make it mono-friendly.
BTW it seems, the result of these changes can be linux and Mac support, right?

sriv commented

However I cannot get that Runner.IntegrationTests assembly to load on windows

Strange, what do you see when you run .\build\test.ps? It should run that test loading the checked-in Gauge.CSharp.Lib.dll. This works on an appveyor build that we maintain parallel to our go.cd pipelines. [Note: the latest appveyor build is broken because of a change in the build scripts, will update now]

generating AssemblyInfo with any version you want v/s just add a strict reference in IntegrationTestSample on old already published package

I think the latter would be a better choice, there is no reason to build the dll fresh, with new versioninfo, this is more like reference data for the test hence, I would like it to be a static entity. What do you say?

I will start working on fixing linux tests and approach the GaugeProjectBuilder to make it mono-friendly.

Sounds good. Just some context, we initially were invoking msbuild.exe and then moved to using the API via Microsoft.Build namespace. It gave us more granular information about the build outcome and console logging.

the result of these changes can be linux and Mac support, right?

I believe so, at least it will be the bulk of what is required, we can iron out anything else that pops up.

On a side note, I just wanted to highlight that there is a CLA in place for Gauge, I hope you are OK to sign it?

tomzo commented

Strange, what do you see when you run .\build\test.ps?

I don't run the powershell scripts, since I presume we want to move away from them. But that script does nothing more than runs nunit-console.exe with test assemblies paths, right?
Now there fake task, that does the same. You can try it out from my fake branch. Just run

build.cmd CopyBinaries # here you should look what is in artifacts/ directory now
build.cmd RunTests # this will fail on integration tests in windows

I have already posted output from #74 (comment)
Quick google search says that these are issues with loading 2 version of the same assembly. Which makes some sense, the Sandbox role is to load second application domain with older Lib, the one in gauge-bin, right? But the problem here, seems to be that integration test is trying to load 0.0.0 version in the test runners domain.

I would like it to be a static entity. What do you say?

OK. I'd like to get tests working first, "the old way" before breaking more stuff anyway.

CLA

Sure. Signed.

New questions

Are you sure this is correct? Shouldn't it be

sandboxDomain.AssemblyResolve += CurrentDomain_AssemblyResolve;

We want the sandbox domain to load assemblies from gauge bin directory. Not the main main domain where runner is executing. Or I misunderstood something...

Case sensitive environment variables

What is gauge statement about environment variables casing?

On linux there are failing tests because in test code there is

Environment.SetEnvironmentVariable("GAUGE_ADDITIONAL_LIBS", "foo.dll");

And then in runner project we are reading

Environment.GetEnvironmentVariable("gauge_additional_libs");

This may have worked on windows, but it won't on linux. My proposition is to make all environment variable reading case insensitive. That means replace all calls to GetEnvironmentVariable by a method which wil first check "GAUGE_ADDITIONAL_LIBS", then "gauge_additional_libs". So that use can set either one of them to get desired result.
Another option is to just use upper case, always. I can do that too. Tell me what you think.

sriv commented

Thanks for the update, I am travelling for the rest of this week starting tomorrow evening, so may not be able to contribute much, apologies if this blocks your progress.

Now there fake task, that does the same. You can try it out from my fake branch.

I will try that out in a bit and respond.

resolving assemblies for SandboxDomain

I thought so too, but it didn't seem to work for me. I will have another look. Logically what you say makes sense, but I remember changing that caused the runner to crash. I won't be surprised if this is incorrect.

the Sandbox role is to load second application domain with older Lib, the one in gauge-bin, right?

Yes, the functional role of the Sandbox is to provide isolation between the test runner and user code's referenced assemblies. Ultimately, a test runner referencing Gauge.CSharp.Lib.dll version 0.5.1 should be able to execute test code referencing Gauge.CSharp.Lib.dll version 0.5.3.

Also if you are changing/refactoring/thinking about the appdomains, please can you also look at #37? I am trying to find a clean way of honouring assembly binding-redirects.

Case sensitive environment variables

Agreed. Environment variables should be case sensitive and upper cased. I am happy to have gauge-csharp take care of case while reading the env vars. I think having a consistent convention that would work cross platform makes sense. Thanks for bringing this out.

tomzo commented

Thanks for the update, I am travelling for the rest of this week starting tomorrow evening, so may not be able to contribute much, apologies if this blocks your progress.

I should be fine as long as you take a look from time to time and respond to some questions.

I thought so too, but it didn't seem to work for me. I will have another look. Logically what you say makes sense, but I remember changing that caused the runner to crash. I won't be surprised if this is incorrect.

IMHO currently the sandboxing is broken and never worked. I would speculate that only assembly that was loaded during integration tests so far was the one with 0.0.0 version, it was loaded in current and sandboxed domain. Assert is that there 0.0.0 loaded in sandbox, which was true. But there is check that test run has non 0.0.0 version. I think we can verify this on master by adding assert.

Also if you are changing/refactoring/thinking about the appdomains, please can you also look at #37? I am trying to find a clean way of honouring assembly binding-redirects.

I might have to, the SandBoxTests are passing when I execute single case, but some are failing when I run them all at once. This happens on linux.

tomzo commented

I finally have progress on loading assembly problems. Above clue was correct, basically the main domain (runner's domain) is at some point is loading (or at least trying to load) the users assembly. In tests that is IntegrationTestSample and Gauge.CSharp.Lib version 0.0.0.
And just to clarify, in .NET it is allowed to have 2 or more versions of the same assembly loaded into the same domain. But from my experience that will have horrible consequences for debugging and maintaince.
I got all tests working in mono first, I replaced that assembly resolution code, so that users methods are loaded only in the sandbox domain. Good, right?
But then I started working on windows and I apprently windows still hates me:

09:28:25.540 1) Test Error : Gauge.CSharp.Runner.IntegrationTests.SandBoxTests.ShouldGetAfterScenarioHooks
09:28:25.540    System.IO.FileLoadException : Could not load file or assembly 'Gauge.CSharp.Lib, Version=0.0.0.0, Culture=neutral, PublicKeyToken=3f92af01a0ea350e' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
09:28:25.540 
09:28:25.540 Server stack trace: 
09:28:25.540    at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
09:28:25.540    at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
09:28:25.540    at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection)
09:28:25.540    at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
09:28:25.540    at System.Reflection.Assembly.Load(String assemblyString)
09:28:25.540    at System.UnitySerializationHolder.GetRealObject(StreamingContext context)
09:28:25.540    at System.Runtime.Serialization.ObjectManager.ResolveObjectReference(ObjectHolder holder)
09:28:25.540    at System.Runtime.Serialization.ObjectManager.DoFixups()
09:28:25.540    at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
09:28:25.540    at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
09:28:25.540    at System.Runtime.Remoting.Channels.CrossAppDomainSerializer.DeserializeObject(MemoryStream stm)
09:28:25.540    at System.Runtime.Remoting.Messaging.SmuggledMethodReturnMessage.FixupForNewAppDomain()
09:28:25.540    at System.Runtime.Remoting.Channels.CrossAppDomainSink.SyncProcessMessage(IMessage reqMsg)
09:28:25.540 
09:28:25.540 Exception rethrown at [0]: 
09:28:25.540    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
09:28:25.540    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
09:28:25.540    at Gauge.CSharp.Runner.ISandbox.GetHookRegistry()
09:28:25.540    at Gauge.CSharp.Runner.IntegrationTests.SandBoxTests.ShouldGetAfterScenarioHooks() in c:\GoAgent\pipelines\gauge-csharp\Runner.IntegrationTests\SandBoxTests.cs:line 146
09:28:25.540

The important part is that it throws on Gauge.CSharp.Runner.ISandbox.GetHookRegistry(). Why? Because there are elements of System.Reflection in the ISandbox which is used as contract in communication between the 2 domains.

public interface ISandbox
    {
        ExecutionResult ExecuteMethod(MethodInfo method, params object[] args);
        bool TryScreenCapture(out byte[] screenShotBytes);
        IHookRegistry GetHookRegistry();
        List<MethodInfo> GetStepMethods();
        void InitializeDataStore(string dataStoreType);
        IEnumerable<string> GetStepTexts(MethodInfo stepMethod);
        List<string> GetAllStepTexts();
        Assembly TargetLibAssembly { get; }
        void ClearObjectCache();
        IEnumerable<string> GetAllPendingMessages();
    }

In short, methods like List<string> GetAllStepTexts(); are OK because types are all primitives or contain only serializable system types. But method like List<MethodInfo> GetStepMethods(); contains ultra-deep object MethodInfo which contains Assembly instances, which eventually has instance Gauge.CSharp.Lib version 0.0.0. So when runner calls Gauge.CSharp.Runner.ISandbox.GetHookRegistry() on the remoting proxy, and Gauge.CSharp.Lib version 0.0.0 is returned then runner's side tries to load that assembly (only on windows), which fails because this assembly is in gauge-bin and I am executing tests from artifacts/gauge-csharp/itest where there is only Gauge.CSharp.Lib version 0.5.3. So assembly loading fails with

 The located assembly's manifest definition does not match the assembly reference

because it has loaded 0.5.3 version instead of 0.0.0 which was returned by ISandbox instance.

A quick solution is to just let the users assemblies be loaded in runners, which is what I did here. So now when proxy returns something forcing Gauge.CSharp.Lib version 0.0.0 to load, then this method will find the 0.0.0 assembly from gauge-bin directory.
In the end, tests are passing on linux and windows now. Hurray!

But I don't think this is correct solution. I think we need to change ISandbox contract into something else. I don't think it should return MethodInfo elements, intead we should ask it to execute some of the runners routines. I don't know much about runner's or gauge's logic so I won't make more suggestions here. @sriv please think about it.

tomzo commented

For those who want try out running gauge C# on linux or mac, you can try out from my branch.

To build and install from source:

git clone --branch mono-pr https://github.com/tomzo/gauge-csharp.git
cd gauge-csharp/
./build.sh Zip
gauge --install csharp -f artifacts/gauge-csharp/gauge-csharp-*.zip

Quick try out

mkdir -p /tmp/gauge-demo &&\
  cd /tmp/gauge-demo &&\
  gauge --init csharp &&\
  gauge specs/

You should see

create  manifest.json
 create  specs
 create  specs/example.spec
 create  env
 create  env/default
 create  env/default/default.properties
 create  gauge-demo
 create  Properties/AssemblyInfo.cs
 create  env/default/csharp.properties
 create  StepImplementation.cs
 create  packages.config
 create  GaugeDemo.csproj
 create  GaugeDemo.sln
 Installing Nuget Package : FluentAssertions, version: 3.5.0
 Installing Nuget Package : Gauge.CSharp.Lib, version: 0.5.3
 Done Installing Nuget Package!
Successfully initialized the project. Run specifications with "gauge specs/"

Building project: /tmp/gauge-demo/GaugeDemo.csproj
  xbuild  /tmp/gauge-demo/GaugeDemo.csproj /t:Build  /v:q  /p:RestorePackages="False" /p:Configuration="Release" /p:Platform="Any CPU" /p:OutputPath="/tmp/gauge-demo/gauge-bin" /logger:Fake.MsBuildLogger+ErrorLogger,"/home/ide/.gauge/plugins/csharp/0.8.0/bin/FakeLib.dll"
xbuild  /tmp/gauge-demo/GaugeDemo.csproj /t:Build  /v:q  /p:RestorePackages="False" /p:Configuration="Release" /p:Platform="Any CPU" /p:OutputPath="/tmp/gauge-demo/gauge-bin" /logger:Fake.MsBuildLogger+ErrorLogger,"/home/ide/.gauge/plugins/csharp/0.8.0/bin/FakeLib.dll"
XBuild Engine Version 12.0
Mono, Version 4.2.2.0
Copyright (C) 2005-2013 Various Mono authors
Gauge.CSharp.Runner.SandboxFactory Creating a Sandbox in: /home/ide/.gauge/plugins/csharp/0.8.0/bin/
# Specification Heading
  ## Vowel counts in single word     ✔ ✔
  ## Vowel counts in multiple word   ✔ ✔

Successfully generated html-report to => /tmp/gauge-demo/reports/html-report
Specifications: 1 executed  1 passed    0 failed    0 skipped
Scenarios:  2 executed  2 passed    0 failed    0 skipped

Total time taken: 1.122s
sriv commented

Now available via release 0.8.0