Building the project in TCXaeShell does not collect test results correctly when generating the XML file.
Traavaas opened this issue · 21 comments
TwinCAT3 Version: 3.1.4024.10
Visual Studio Development Tools Environment (DTE) version: TcXaeShell.DTE.15.0
TcUnit Version: 1.1.0.0
TcUnit-Runner Version: 0.9.3.0
When building the project in Visual Studio 2013, the XML file looks right. This is what it looks like:
<?xml version="1.0" encoding="utf-8"?>
<testsuites failures="0" tests="4">
<testsuite id="0" name="PRG_MAIN.fb_Subtract_Test" tests="2" failures="0">
<testcase name="FiveMinusThreeEqualsTwo" assertions="1" classname="PRG_MAIN.fb_Subtract_Test" status="PASS" />
<testcase name="TwelveMinusFiveEqualsSeven" assertions="1" classname="PRG_MAIN.fb_Subtract_Test" status="PASS" />
</testsuite>
<testsuite id="1" name="PRG_MAIN.fb_Sum" tests="2" failures="0">
<testcase name="TwoPlusTwoEqualsFour" assertions="1" classname="PRG_MAIN.fb_Sum" status="PASS" />
<testcase name="ZeroPlusZeroEqualsZero" assertions="1" classname="PRG_MAIN.fb_Sum" status="PASS" />
</testsuite>
</testsuites>
However, when it tries building in TCXaeShell, the XML file looks like this:
<?xml version="1.0" encoding="utf-8"?>
<testsuites failures="0" tests="4" />
When it tries to record that file in Jenkins, My pipeline outputs this:
[Pipeline] junit
Recording test results
None of the test reports contained any result
When I build and run the project in TCXaeShell itself (away from Jenkins and TcUnitRunner), I am able to see the test cases and details in the output and Error List.
Thanks for your report @Traavaas.
Is it possible for you to upload the project here? I'll try to reproduce on my end.
@sagatowski Appreciate your help.
@Traavaas Which version of TcUnit are your libraries/TcProjects using? 1.1 or the current trunk?
Also, which version of TcUnit-Runner?
Updated original post with those details!
Hey @Traavaas!
I have now done the following:
Created a fresh new machine, with:
- Windows 10,
- installed Tc 3.1.4024.10,
- TcUnit-Runner 0.9.3,
- TcUnit 1.1
- and your example code.
I noticed there were two tasks in the example project, so TcUnit-Runner needs to know which one to run (if there is only one it will run that one), so this needs to be provided as input argument (it's described in the documentation if you want to know all details). Note the -t PlcTask argument provided
I executed TcUnit-Runner in this way and got the following result:
C:\Program` Files (x86)\TcUnit-Runner>TcUnit-Runner.exe -v "C:\Code\SampleProject\SimpleExampleProject\SimpleTcUnitExampleProject.sln" -t PlcTask
2021-03-24 19:27:56 - TcUnit-Runner build: 0.9.3.0
2021-03-24 19:27:56 - TcUnit-Runner build date: 2020-12-22
2021-03-24 19:27:56 - Visual Studio solution path: C:\Code\SampleProject\SimpleExampleProject\SimpleTcUnitExampleProject.sln
2021-03-24 19:27:56 -
2021-03-24 19:27:56 - In TwinCAT project file, found TwinCAT version 3.1.4024.10
2021-03-24 19:27:56 - In Visual Studio solution file, found visual studio version 15.0
2021-03-24 19:27:56 - Version is pinned: False
2021-03-24 19:27:56 - Trying to load the Visual Studio Development Tools Environment (DTE) version 'TcXaeShell.DTE.15.0' ...
2021-03-24 19:28:03 - ...SUCCESSFUL!
2021-03-24 19:28:15 - Using the TwinCAT remote manager to load TwinCAT version '3.1.4024.10'...
2021-03-24 19:28:33 - Setting task 'PlcTask' enable and autostart, and all other tasks (if existing) to disable and non-autostart
2021-03-24 19:28:57 - Setting target NetId to '127.0.0.1.1.1'
2021-03-24 19:28:57 - Enabling boot project and setting BootProjectAutostart on 127.0.0.1.1.1
2021-03-24 19:29:43 - Waiting for results from TcUnit...
2021-03-24 19:29:59 - ... got 32 report lines so far.
2021-03-24 19:29:59 - All results from TcUnit obtained
2021-03-24 19:30:14 - Done collecting TC results
2021-03-24 19:30:14 - Writing xUnit XML file to C:\Code\SampleProject\SimpleExampleProject\TcUnit_xUnit_results.xml
2021-03-24 19:30:14 - Closing the Visual Studio Development Tools Environment (DTE)...
2021-03-24 19:30:44 - Exiting application...
And this is the final end-result (in the xml-file)
<?xml version="1.0" encoding="utf-8"?>
<testsuites failures="0" tests="4">
<testsuite id="0" name="PRG_TEST.fb_Subtract_Test" tests="2" failures="0">
<testcase name="FiveMinusThreeEqualsTwo" assertions="1" classname="PRG_TEST.fb_Subtract_Test" status="PASS" />
<testcase name="TwelveMinusFiveEqualsSeven" assertions="1" classname="PRG_TEST.fb_Subtract_Test" status="PASS" />
</testsuite>
<testsuite id="1" name="PRG_TEST.fb_Sum" tests="2" failures="0">
<testcase name="TwoPlusTwoEqualsFour" assertions="1" classname="PRG_TEST.fb_Sum" status="PASS" />
<testcase name="ZeroPlusZeroEqualsZero" assertions="1" classname="PRG_TEST.fb_Sum" status="PASS" />
</testsuite>
</testsuites>
In other words, all worked perfectly fine.
Did you do anything different than above?
I've had a long day at work today so I will go to sleep now, but if you do any findings please write them down here and I'll try to answer as soon as I can.
/ J
I executed the program the same way you did.
C:\Program Files (x86)\TcUnit-Runner>TcUnit-Runner.exe -v "C:\Users\tyu\tcunitargumentstest\SimpleExampleProject\SimpleTcUnitExampleProject.sln" -t PlcTask
2021-03-24 19:49:44 - TcUnit-Runner build: 0.9.3.0
2021-03-24 19:49:44 - TcUnit-Runner build date: 12/22/2020
2021-03-24 19:49:44 - Visual Studio solution path: C:\Users\tyu\tcunitargumentstest\SimpleExampleProject\SimpleTcUnitExampleProject.sln
2021-03-24 19:49:44 -
2021-03-24 19:49:44 - In TwinCAT project file, found TwinCAT version 3.1.4024.10
2021-03-24 19:49:44 - In Visual Studio solution file, found visual studio version 15.0
2021-03-24 19:49:44 - Version is pinned: False
2021-03-24 19:49:44 - Trying to load the Visual Studio Development Tools Environment (DTE) version 'TcXaeShell.DTE.15.0' ...
2021-03-24 19:49:46 - ...SUCCESSFUL!
2021-03-24 19:49:49 - Using the TwinCAT remote manager to load TwinCAT version '3.1.4024.10'...
2021-03-24 19:49:56 - Setting task 'PlcTask' enable and autostart, and all other tasks (if existing) to disable and non-autostart
2021-03-24 19:50:06 - Setting target NetId to '127.0.0.1.1.1'
2021-03-24 19:50:06 - Enabling boot project and setting BootProjectAutostart on 127.0.0.1.1.1
2021-03-24 19:50:35 - Waiting for results from TcUnit...
2021-03-24 19:50:45 - ... got 6 report lines so far.
2021-03-24 19:50:45 - All results from TcUnit obtained
2021-03-24 19:50:55 - Done collecting TC results
2021-03-24 19:50:55 - Writing xUnit XML file to C:\Users\tyu\tcunitargumentstest\SimpleExampleProject\TcUnit_xUnit_results.xml
2021-03-24 19:50:55 - Closing the Visual Studio Development Tools Environment (DTE)...
2021-03-24 19:51:15 - Exiting application...
My xml file looks like this:
<?xml version="1.0" encoding="utf-8"?>
<testsuites failures="0" tests="4" />
Any suggestions?
@Traavaas That is really strange. Strange indeed. It's as if it only sees the final test results (how many tests have been running and failed, succeeded etc), but not the individual test results.
I have managed to re-produce your error, but then I had to manually go into the project and change this parameter To FALSE:
But it has to be TRUE, which is the default behavior, so I guess you have not changed this to FALSE?
If I set the above to FALSE, then I get this:
<?xml version="1.0" encoding="utf-8"?>
<testsuites failures="0" tests="4" />
Also, do you have any other versions of TwinCAT installed?
You installed the version 1.1 from RELEASES?
Just trying to understand what differs between our machines now...
Yeah, I have not messed with TcUnit itself. Mine looks exactly like yours.
I do not have any other versions of TwinCAT installed. I definitely installed the version from RELEASES. I compared the sha-1 of the compiled-library of the one I have on my computer to the one I just download now, and they happen to be the same.
@Traavaas Also, if you run the project manually in TcXaeShell, do you get the following messages marked in red?
The messages are the ones that are used to gather information on test results
I do get the messages marked in red.
Everything looks in order... so TcUnit-Runner should get all results and all should be fine.
I will have to sleep on this one and think about it.
I might add some verbose debug logs to the TcUnit-Runner and do a special-compile for you (or I might simply add a -verbose mode to the logger), and then you could return the output to me?
Yeah that sounds good. Thanks again for your help!
@Traavaas Enabling debug-level in log4net on the fly was a little trickier than I thought. It will have to be a weekend-project. I'll keep you updated.
Alright, I have implemented a debug level of logging. Actually re-did many parts of the logging while I was at it. I'm much more happy with the logging now 😄
Options:
-v, --VisualStudioSolutionFilePath=VALUE
The full path to the TwinCAT project (sln-file)
-t, --TcUnitTaskName=VALUE [OPTIONAL] The name of the task running TcUnit
defined under "Tasks"
-a, --AmsNetId=VALUE [OPTIONAL] The AMS NetId of the device of where
the project and TcUnit should run
-w, --TcVersion=VALUE [OPTIONAL] The TwinCAT version to be used to
load the TwinCAT project
-u, --Timeout=VALUE [OPTIONAL] Timeout the process with an error
after X minutes
-d, --debug [OPTIONAL] Increase debug message verbosity
-?, -h, --help
So simply add -d when executing. I haven't made a release of this yet, so simply build TcUnit-Runner from source.
Run the project again (with -d), and upload the output file (in /logs) from that run here.
Attached is my output from the run of your TwinCAT solution.
2021-03-27_09_27_24.log
Can you explain how to build TcUnit-Runner from source? I've cloned the repository to my computer so far.
Can you explain how to build TcUnit-Runner from source? I've cloned the repository to my computer so far.
It's described on the main page: https://github.com/tcunit/TcUnit-Runner
Go to "Want to contribute?"
@sagatowski Thanks for taking the time to help me! I was able to to build and run the project with the -d argument perfectly. I'm not sure what exactly changed but after building and running from source I ran my project with the release version and it happened to work perfectly as well. It was also able to record the results in Jenkins. I was also able to verify the the results are correct in the xml file. This is very odd because I did not make a single change to the project over the weekend. I believe this may be a problem of my computer rather than TcUnitRunner.
That's weird! At what device were you running the TcUnit-Runner? VirtualBox? Your PC? PLC?
Do you have any idea what the issue with the computer might have been?
I'm using my PC. I've been trying to figure out the reason why it randomly started working and I have no clue. I re-downloaded the zip file I uploaded to you and ran tcunit-runner (The release version that I have always been using without the new debug feature) on it. It ended up working correctly! It makes no sense to me haha
Alright, one time is no time. I'll close this issue.
If you encounter this problem again, upload a debug log here and we'll re-open it.