Add custom NUnit Console options (arguments)
Opened this issue · 12 comments
Hi!
Generally speaking this is not an issue, but a feature request
The idea is simple:
- Support custom console attributes
- Provide the way to access nunit3-console arguments inside test engine extensions
This feature can be very useful for extending NUnit Test Engine.
@elv1s42 GitHub issues are how we get feature requests as well as bug reports, so that's fine.
However, it's not clear what you are asking for. What do you mean by custom console attributes?
WRT access to arguments by extensions, perhaps you can give an example of what you would like to do.
@CharliePoole Sorry, I did not provide enough details of this feature request.
What I want is to be able to get console arguments inside Engine extensions. For example, when I run tests with console:
nunit3-console.exe C:\Tests\MyTests.dll -agents=1 -output="C:\out.txt" -work="C:\NUnitResults"
I want to get this values inside my extension. Something like this:
[Extension(Description = "Test Reporter Extension", EngineVersion = "3.4")]
public class MyEventListener : ITestEventListener
{
public void OnTestEvent(string report)
{
var xmlNode = XmlHelper.CreateXmlNode(report);
switch (xmlNode.Name)
{
case "test-run":
{
//I want to get '-work=...' value for my reporting tool:
var workPath = NUnitConsole.GetArgValue("work");
//And save report into workPath:
ReportingTool.GenerateReport(xmlNode, workPath);
break;
}
}
}
}Another idea is to support "user defined" arguments:
nunit3-console.exe C:\Tests\MyTests.dll -myArgument=myValue
Do you think this is useful? Is it possible to implement this feature?
Hi! Any updates on this request?
Allowing Engine extensions to access console arguments is a bit of a tricky concept. The engine has no idea whether it is being run by the console runner, gui runner, vs adapter, etc. That's part of the overall layered architecture of NUnit.
However, in the example you give, you do have a reasonable request. You want the extension to be able to use the work directory that the engine is using. Ideally, you would want to know that no matter how that work directory is set... command-line argument, default nunit setting, value in a project file, value in a run settings file, etc.
I think we should look at this request as intending that the extension should be able to get information from the engine under which it is running rather than from the command-line. @elv1s42 would that suit your needs?
@CharliePoole this is even better! Your suggestion will definetly suit the need from example.
I didn't new that Engine uses options coming from comand-line args, settings file or project file. I think this feature request can be changed, the idea of reading engine options (like work directory) from extensions looks interesting.
If there is no separate repository for NUnit Engine I guess we don't need to move this request to another repo.
@elvis42 I'll take a look at the interfaces a little later and add some comments about how this might be done without breaking compatibility for existing extensions.
+1 for this - I'd like to do the same (logging errors out to a file for each test, tests running in parallel).
This one comes up now and then, and particularly ties into something I'm looking at at the moment. I'd like to take this issue forward at some point, so here's a rough shot at a design.
My aim is to be able to have extensions define arguments and receive values. Runners should be able to continue to validate arguments as they do now, and not have to lessen the current validation in case of potential unknown arguments. It of course should be possible for arguments to be defined without prior engine knowledge (e.g. the current --teamcity option), and the feature must be usable by all different types of runner e.g. command line, GUI, VS adapter.
Thoughts on the below welcome. 🙂 cc @nunit/engine-team
Extensions
Extensions would define any arguments they require in an attribute. Each argument is defined by a short-name, display-name, type and description. To receive options, extensions would implement an IArgumentReceiver interface. (Better name required...)
[Extension]
[ExtensionArgument("logFileName", "Log File Name", typeof(string), "The file path to the logging file.")]
public MyLoggingExtension : ITestEventListener, IArgumentReceiver
{
private string _logFilePath;
public void ReceiveArguments(IArgumentsService argsService)
{
_logFilePath = argsService.GetOption<string>("logFileName");
}
/*...*/
}For arguments that can be supplied in multiple, array types should be used - and enums for arguments with limited permitted values.
Currently extensions are not initialised until required, so the guarantee would need to be that ReceiveArguments() is called immediately after extension initialisation. I'm not yet sure how difficult that would be to implement.
ExtensionService
When discovering extensions, the Extension Service would also become responsible for identifying any ExtensionArgument attributes, and registering the arguments with the ArgumentsService.
argumentsService.RegisterArg("logFileName", "Log File Name", typeof(string), "The file path to the logging file.");Runners
Runners that support this feature would request an initialised IArgumentsService from the engine. From the IArgumentsService the runner can request the list of all other known argument definitions, and permit user input as required.
e.g.
- The Console could add additional valid command line options
- The Adapter could allow different xml values in the .runsettings file
- A GUI runner could dynamically create additional type-specific UI menus.
ITestEngine engine = TestEngineActivator.CreateInstance();
IArgumentsService argsService = engine.Services.GetService<IArgumentsService>();
IArgument[] args = argsService.GetAllDefinedArguments();
foreach(var arg in args)
{
consoleOptions.Add($"{arg.shortName}=", arg.Description,
v => arg.Value = parser.RequiredValue(v, "--logFilePath"));
}Any options defined by the user would then be set back on the IArgumentsService, to later be passed on to the extension itself.
foreach(var arg in args.Where(a => a.Value != null)
{
argsService.DefineOption(arg);
}Random thoughts off the top of my head...
Nice idea!
Seems slightly related to extension properties, which are currently readonly and used by the extension to tell the runner more specifically what it does. For example, result writer extensions say what format names they support.
The only existing settable property we have is Enabled. Could that be generalized?
The extension node is the point of communication. Right now the extension can't access it, but maybe that should be provided. It could then serve as a blackboard for communicating both ways.
It's valuable to be able to get information about an extension without loading it. Again, using result writers as an example, there's no need to actually load one unless the user asks for it. Ideally, we should keep that ability for arguments, allowing them to be saved in the extension node and then only retrieved if the extension is actually being loaded.
I wonder if we could simplify this by making everything a string. The user has to enter strings on the command-line and in the runsettings file anyway. It might be up to the extension to interpret the string. This is of course what we already do with most settings for both the engine and framework anyway.
Thanks Charlie. 🙂
Seems slightly related to extension properties, which are currently readonly and used by the extension to tell the runner more specifically what it does. For example, result writer extensions say what format names they support. The only existing settable property we have is Enabled. Could that be generalized?
Interesting thinking, I didn't know "Enabled" was implemented as a settable properly, I thought they were all readonly. I'll have a think about that...
The extension node is the point of communication. Right now the extension can't access it, but maybe that should be provided. It could then serve as a blackboard for communicating both ways. It's valuable to be able to get information about an extension without loading it. Again, using result writers as an example, there's no need to actually load one unless the user asks for it. Ideally, we should keep that ability for arguments, allowing them to be saved in the extension node and then only retrieved if the extension is actually being loaded.
Absolutely agree continuing to not load extensions unless required. Not so sure I'm quite following the idea of using the ExtensionNode as a two-way communication object. Any chance of some pseudo-code on the sort of thing you're thinking?
I wonder if we could simplify this by making everything a string. The user has to enter strings on the command-line and in the runsettings file anyway. It might be up to the extension to interpret the string. This is of course what we already do with most settings for both the engine and framework anyway.
I wanted to make it type-safe so the runners can continue being responsible for (most) argument validation. For example, for an argument of type bool named "DoLogging", the console could accept "true" and "false" but not "hedgehog". A GUI runner may chose instead to display a checkbox rather than a text-input. XML settings like .runsettings may chose to recognise a <DoLogging /> element rather than requiring <DoLogging>true</DoLogging>. Personally, for a small additional cost, I think it provides worthwhile benefit.
Well, to be fully accurate, Enabled is not a property of the extension but of the ExtensionNode. The extension doesn't need to know it's enabled/disabled because it only runs when it's enabled.
But then all properties are properties of the Extension Node because that's what the runner deals with in the abstract. Some are read-only and some can be set. It's only after the user accesses the ExtensionObject property that the extension is actually initialized. It could be passed arguments at the time of initialization, or just passed an interface to its extension node.
I do see the benefit of type-safe arguments or property values, I don't see the effort as small is all. 😄
Cleaner to do this for 4.0, where we are allowed to make breaking changes!