Any chance of adding a global setup/global teardown
stagg54 opened this issue · 12 comments
VI Tester had a global setup/teardown that would execute once per test case.
Any plans to implement something similar?
Barring that, is there some sort of workaround?
Hey,
I am not sure I understand completely what you are requesting. In xUnit terminology, a "test case" is one test, it is implemented as an instance of a test case class at runtime. So each class would contain many test methods and when the tests are run for all methods in the class, one instance of the test case class is created for each method. So using this terminology, the setup/teardown VI:s would be global.
But I assume you would like some additional setup/teardown running for each test execution? Or what would be the chunk size: at the beginning of and end of each execution of all tests belonging to a specific test case class? Something like this:
global setup
setup
test case 1
tear down
setup
test case 2
tear down
global teardown
Could you explain your use case a bit more? If you are working on tooling, there could be better hooks for you to use and many of the features in LUnit are actually plugins, although I've never documented how it works,
In general it feels like an antipattern to spread out the test logic over many different files as the tests get very obscure to read. I tend to prefer using helper methods in my tests explicitly over setup/teardown, but there are cases where setup/teardown fits better.
Maybe this helps.
Ok, thanks. That explains a lot.
LUnit follows a slightly different model, something like this:
http://xunitpatterns.com/Testcase%20Class.html
So here it is not clear when a global setup/teardown would execute. It does not really fit the model.
Could you explain your use case? There might be other hooks you can use.
Meszaros would refer to that as a test fixture.
http://xunitpatterns.com/SuiteFixture%20Setup.html
PyTest does something similar.
My understanding is this is a pretty common unit testing and xunit pattern.
My use case is that I have some code (for scripting) that traverses a directory hierarchy and parses some information. I would like to create a directory hierarchy for all my tests to parse in a temp directory. I've got several tests for this code. It is non-destructive, so they can all share the same test directory hierarchy. I'd like to create the hierarchy just once instead of having to create it for each test.
It seems like a rather common scenario - many tests share setup code that is the same between each test. Yes you could do that as a utility method that runs at the beginning of each test method.
I can agree that moving test code out of the test methods can obscure things.
However there are 2 issues with the utility method idea:
- you have to remember to add it to the beginning of every test method.
- you end up running that code multiple times. In this case it touches the file system so it is expensive and it doesn't need to be run every time, yes I guess I could add a check to see if it already exists, but that feels cumbersome.
Having a predefined hook to run this type of code seems to be the norm in other testing libraries. Putting in the predefined place should make it obvious what is going on and make it less obscure.
actually I take back part of that.
Looking closer at the link I shared, Meszaros would call that an Implicit setup - executes once per test case, versus an explicit setup that exercise once per test method.
He refers to a fixture as something that executes once per suite.
The terminology is rather cumbersome.
Pytest just refers to them all as fixtures, regardless of when the execute. Each has a scope which determines when it executes.
I would be happy with taking the pytest approach. Not sure how best to do that. Caraya somewhat resembles Pytest, not sure how they implement any of this - not sure that Caraya does.
Actually its been a while since I read that book and the terminology is very confusing.
Perhaps by inline he means part of the 4 part test defined in the method?
I need to pull out the book and look at it again.
That thing is a beast!
The terminology is a bit of a mess and seems to be diverging a lot depending on the language and test framework.
The key point is that in LUnit there is no such thing as a test method at run time. You have a test case class, which has member VIs. It then acts as a factory producing a suite of test case objects which are executed by the runner. Essentially the command pattern.
You thus have a set of test case objects, called a test suite, and these suites may be nested. When executed, suites are distributed to test runners which run each suite in parallel.
So to have a vi tester style global setup for a test case would translate to having a setup for a test suite created from a test case class. Or something similar?
I see how this could be useful and anything making tests run faster is a big win in my book. I'll do some prototyping to see where would be a good place to add a hook.
Touching the file system is always messy in testing and I see why you would not want blow it away and recreate for each test. As long as the tests are not mutating anything you could get away with creating it once, maybe you could just keep it static as part of the test? If it is mutated, it gets more messy..
Yeah that was my thought as a workaround just have a test directory as part of my repo and use that.
Just want to make sure i understand though, the setup and teardown in LUnit - when and how often do they execute?
Do they execute once per test class? or once for each individual test method?
I had assumed they behaved similar to VITester (ie once before and after each test) but it sounds like they execute once for the entire test class.
Sorry for the confusion, it works exactly as you have assumed. See link below. (I noticed that I am not completely consistent in the terminology in the documentation either, partly because the implementation details and my understanding have changed historically)
https://lunit.astemes.com/10_Basics/#using-the-setup-and-teardown-methods
To clarify, what you call a test method would be the same as what I mean by a test case. So the setup would run before each test case, which is a single test as defined in a test*.vi. In LUnit the entities are test case and test suite, there is no test method.
Also, a test case class is never executed. It contains the test*.vi:s and at runtime produces a test suite (factory method), which is a set of test case objects. The test case objects are executed (command pattern) by the runner and the results are collected.
This is why I'm struggling a bit to figure out where to slap on a global setup.
Hey,
I made some refactoring in LUnit and it is now possible (v1.4.x ->) to implement global fixtures. It still is not natively supported (it might get integrated later if there are users).
I made a reference implementation which can be found here:
https://github.com/Astemes/astemes-lunit-global-fixture/
LUnit v1.4.10 is available here:
https://github.com/Astemes/astemes-lunit/releases/tag/v1.4.10
Let me know what you think.
I'll close off this issue for time being.
The reference implementation of global fixtures should provide the needed functionality and we'll consider officially integrating it if there are users.