Test results of nested test suites don't show up in result summary
Opened this issue · 13 comments
Describe the bug
When calling a test suite inside another test suite, the results of that test suite are not added to the result summary
To Reproduce
Make a test suite (A), add another test suite (B) as a var to A. Call test suite B in the body of A.
Expected behavior
Expected behaviour is that all the test results count towards the test result summary
Screenshots
(there's only 1 Assert in the test, different instantiations)
Software versions
TcUnit v1.2.0.0 TwinCAT 4024.12
Run environment
local on PC runtime
Additional context
You would ask, why would you need nested tests? I want to use the same test suite for multiple implementations of an interface. Each implementation requires a different setup, so I want to call the testSuite in the body and input prepared objects.
Why would you call a test suite from a test suite? Tests should be independent from each other.
How else can you prepare objects before testing them? I want to prepare an array of objects, and then call the test suite with the array as input.
Declare the objects in the same test suite (or better yet, in the testcase) where you test the object under test.
Declaring the objects in each test case requires rewriting the whole test for each implementation of an interface. This way, I only have to prepare the objects once for each implementation, and I use the interface in the test.
If I declare the objects in the body of the test suites, I cannot use the same tests for all of the objects, because I could want tests for multiple interfaces in the same test suite. A test suite cannot extend multiple other ones.
Sorry, I don't understand your description. Can you elaborate (with code snippets)?
Every test requires to be independent of each other.
I made a quick proof of concept here https://github.com/Bulkje/TcUnitSandbox
The example doesn't really show my problem fully, imagine needing relations between instances of a FB, and needing to make those relations in different ways. Then you can do that in the prepare method, without exposing other methods to the test.
In my example I can add another implementation of ISum, and keep the tests the same, just have to add a new prepare method.
This also allows me to add IMultiply easily for example.
You seem to have uploaded the wrong example. The tests seem to not even have a TEST() and TEST_FINISHED()-method defined.
Did you read the documentation for how to write tests at the TcUnit.org website?
The test suite that has tests in it is SumTest, it is instantiated in CalculatorTest and BrokenCalculatorTest which are instantiated in MAIN. I cloned the .git again in a new local repository and it works fine for me.
Did you have a chance to look at this more thoroughly?
Hi! Unfortuanetly not, I'm currently overloaded with work. Anyone else in the TcUnit community?
@Bulkje
I get a different behaviour than you have in your screenshots above:
Also, your tests are missing TEST() and TEST_FINISHED().
On top of this, your test-suites are instantiating other test-suites.
A test-suite should not depend on another test-suite, and a test should not depend on another test.
They should be completely independent from each other. So you should not "call a test suite from another test suite". It's not how xUnit frameworks work.
You can however have as many test suites as you want that test (similar) behaviour, but it doesn't mean that you have to link the test-suites to each other but you can simply create them independently from each other. The same applies to tests.
Whoops, must've changed the code a bit since taking the screenshot.
Maybe I'm misunderstanding you. I want to change different implementations of an interface with the same test code.
That means I have to instantiate the test code multiple times, and the main test suite of the FB under test has to instantiate the interface test suite. That seemed the easiest way to implement my idea to me.
Creating a new test suite for each implementation of an interface is doing double work or code duplication. That is, the implementations of the interfaces should perform the same behaviour. (So they should be tested with the same test code).
I'm not familiar with xUnit, how would it handle using the same test code for testing multiple classes / fb's / etc. ?
I want to change different implementations of an interface with the same test code.
So you have some behaviour you want to verify? But you want to implement that behaviour in multiple ways?
Write a test suite for the behaviour you want to test. Then create a test if you have different variants you want to test.
You have to define a starting-point for the test, and an ending-point, for each and one of those variants. Inbetween those you do your assertions. You can't get away from that. If you have some test-fixture that you want to re-use, you can initiatialize it in the test-suite, and use it for the different tests (though I would not recommend this as the tests should not depend on each other and change each others states).