JetBrains/teamcity-cpp

Format output from similar parallel tests

Closed this issue · 2 comments

pps83 commented

We use flowId with multiple tests running in parallel. Some of these parallel tests run very similar commands with different settings, as a result output that we get in TC is very confusing:

image

As you can see there are two Base test suites and it's not clear which one came from what command. In that screenshot we start 10 test runners in parallel. 2 runners for Base testsuite, for another test suite we start 4 test runners, so the output becomes very confusing if any one of them fails and we need to figure out which one of 4.

So, we run 10 different commands and these are the requirements that we have:

  • for each of the 10 tests running in parallel we want to know what command was used to start it (so that you could easily run that exact cmd to run it again).
  • we run identical test executable with different flavor/environment (let's say Flavor1, and Flavor2). The output should be what flavor used to make that output.
  • test aggregate stats for Base.Atomic shouldn't care if it's Flavor1 or Flavor2.

What formatting should we use to fix the output that TC produces? I can wrap entire output of BaseFlavor1 into its own test suite, but then aggregate test stats will be separate from BaseFlavor2.

Please suggest how to format it properly.
Thanks

This is not directly related to teamcity-cpp project and unfortunately I don't know how to format it properly.
Could you ask at support forum at https://teamcity-support.jetbrains.com/hc/en-us/community/topics/200366709-TeamCity-General-Topics ?

pps83 commented

First I tried to wrap outputs into testSuiteStarted/testSuiteEnded, but that affects tests stats aggregation.
Then I changed that to blockOpened / blockClosed and it works for me properly now.