Muted test webhook doesn't include test name

Question

Muted test webhook doesn't include test name

matt-richardson opened this issue 2 years ago · 16 comments

I'm checking out the new test muted and test unmuted events, to see how we can tie this into our other systems.

For unmuted, we get the test name, but for muted, we dont get the test name.

Expected Behavior

buildEventType | TESTS_UNMUTED
mutedOrUnmutedTests | {4=[suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName]}

buildEventType | TESTS_MUTED
mutedOrUnmutedTests | {4=[suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName]}

(I'm not 100% sure what the 4 is about...)

Current Behavior

buildEventType | TESTS_UNMUTED
mutedOrUnmutedTests | {4=[suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName]}

buildEventType | TESTS_MUTED
mutedOrUnmutedTests | {5=[]}

Steps to Reproduce (for bugs)

Create a basic teamcity project, add a command line runner with the following content:

echo "##teamcity[testSuiteStarted name='suiteName']"
echo "##teamcity[testSuiteStarted name='nestedSuiteName']"
echo "##teamcity[testStarted name='package_or_namespace.ClassName.TestName']"
echo "##teamcity[testFailed name='package_or_namespace.ClassName.TestName' message='The number must be 20000' details='junit.framework.AssertionFailedError: expected:<20000> but was:<10000>|n|r    at junit.framework.Assert.fail(Assert.java:47)|n|r    at junit.framework.Assert.failNotEquals(Assert.java:280)|n|r...']"
echo "##teamcity[testFinished name='package_or_namespace.ClassName.TestName']"
echo "##teamcity[testSuiteFinished name='nestedSuiteName']"
echo "##teamcity[testSuiteFinished name='suiteName']"

Add a webhook that triggers on mute and unmute.
Mute / unmute the test.

Your Environment

tcWebHooks Version: 1.2.2
TeamCity Version: 2023.05
TeamCity server Operating System: Linux
Are you using a WebHook Template?: Legacy Webhook (Name Value Pairs)

Investigation so far

Looking at the code, in WebhookListener.java, it creates a new map, but we could potentially change it from

this.processTestEvent(BuildStateEnum.TESTS_MUTED, null, Collections.singletonMap(muteInfo, new ArrayList<>()));

to

this.processTestEvent(BuildStateEnum.TESTS_MUTED, null, Collections.singletonMap(muteInfo, muteInfo.getTests()));

and I think it should work...
(Note that I haven't tested this at all)

Answer 1 · 2023-07-04T06:06:36.000Z

Hi @matt-richardson. Thanks for your detailed report (as per usual).

If I remember correctly, the list of tests was not available for muted.

I'll have a look at your suggestion just in case I missed something obvious.

Answer 2 · 2023-07-04T10:12:34.000Z

Thanks @matt-richardson. It looks like your suggestion works.

I've pushed to a branch here: https://teamcity.jetbrains.com/buildConfiguration/WebHooksAndOtherPlugins_TcWebHooks?branch=issue_219-muted-tests&buildTypeTab=overview&mode=builds

Can you give that a try and I'll look to do so a release later this week if it all goes well.

Answer 3 · 2023-07-04T10:29:22.000Z

Hmm. ~~Looking at the code in that file on line 121, I don't think it ever worked for muted.

It checks for a non-empty list of tests before executing the webhook. School boy error. 🫣~~

Ignore. It's too late. I'm not being sense.

Answer 4 · 2023-07-06T02:58:00.000Z

Happy to report that it works successfully for both muted and unmuted now :success:.

derivedBuildEventType | TESTS_MUTED
mutedOrUnmutedTests | {13=[suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName]}

derivedBuildEventType | TESTS_UNMUTED
mutedOrUnmutedTests | {11=[suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName]}

Answer 5 · 2023-07-06T02:59:40.000Z

~~Another question with this... What is the 13= and 11= on these?~~

~~I'm using the legacy / deprecated name value pairs format to get these, so it may just be a side affect of that, but not entirely sure how they'll format in json?~~

Nevermind: figured it out... It's the default .toString() implementation of MuteInfo - it just renders the id.

Answer 6 · 2023-07-06T03:14:01.000Z

I imagine it the result of calling toString on that value.

This is what I get.

I am doing the following in velocity

## Define macro called "showTests"
#macro( showTests $myTests)
#if ( $myTests.size() > 0 ) 
#foreach( $tests in $myTests.keySet() )##
#foreach( $test in $tests.getTests() )##
• #escapejson($test.name)
#end
#end
#else No Changes found #end
#end

It prints the bulleted list in the following screenshot when posted to Slack.

Answer 7 · 2023-07-06T03:15:10.000Z

I am looking at exactly that file right now :)

Answer 8 · 2023-07-06T23:43:52.000Z

I've successfully got this all working to my needs - I'm now able to get hooks with:

{
  "event": "TESTS_UNMUTED",
  "tests": [
    {
      "name": "suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName",
      "id": "-8186798926076262107",
      "user": "matt@example.com"
    }
  ]
}

and

{
  "event": "TESTS_MUTED",
  "tests": [
    {
      "name": "suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName",
      "id": "-8186798926076262107",
      "user": "matt@example.com"
    }
  ]
}

For posterity, this is based on the following template:

{
    "event": "$buildEventType",
    "tests": [
#foreach( $entry in $mutedOrUnmutedTests.entrySet() )##
#foreach( $test in $entry.value )##
      {
        "name": "#escapejson($test.name)",
        "id": "$test.testNameId",
        "user": "$entry.key.mutingUser.username"
      }#if( $foreach.hasNext ),#end
#end
#end
    ]
}

Answer 9 · 2023-07-06T23:46:08.000Z

Nice work. Thanks for the update. I'll get a new release out over the weekend. Thanks for your help on this.

Answer 10 · 2023-07-07T04:51:09.000Z

@matt-richardson If you get a chance, you could pop those into https://github.com/tcplugins/tcWebHooks/wiki/WebHook-Templates-%3A-The-Velocity-Templating-Engine

Answer 11 · 2023-07-08T22:12:38.000Z

Good call - I've updated the wiki 👍

Looks like 1.2.3 is out - I'll give that a go.

Answer 12 · 2023-07-09T01:50:09.000Z

Very nice. I like the foreach.hasNext to conditionally add the closing comma.

Thanks again. I think we can close this ticket.

Answer 13 · 2023-07-09T01:53:44.000Z

FYI. If you're interested in sharing a working template set, there is a project here: https://github.com/tcplugins/tcWebHooksTemplates

I really really need to update the instructions on that repo for exporting and importing templates, as it's all possible via the GUI now. No more curl and crazy REST malarky.

Answer 14 · 2023-07-11T04:30:21.000Z

I've updated the instructions at https://github.com/tcplugins/tcWebHooksTemplates

Answer 15 · 2023-07-18T02:19:12.000Z

I know I shouldn't add onto a closed issue, but.... i'm going to anyway 😂

Im seeing a bunch of errors:

[2023-07-18 02:16:17,787] ERROR [@7438424f'; Normal executor 28] - jetbrains.buildServer.SERVER - AbstractWebHookExecutor :: trackingId: dadfab32-86b1-4bab-8c1a-afd38e6870e4 :: projectId: OctopusDeploy_OctopusServer_CorePlatform :: webhookId: id_905369674 :: templateId: buildmonitormutes, errorCode: 902, errorMessage: Template 'buildmonitormutes' does not support build state 'beforeBuildFinish'

The template is configured like this:

But the webhook is showing all the boxes as ticked, but disabled:

Am I holding it wrong?

Answer 16 · 2023-07-18T02:26:15.000Z

Looks like a UI issue. Can you please raise as a new ticket?
I think it's because the template was re-selected in the UI from one that did support all event types, hence are selected but disabled.

You might be able to work around by opening and saving and opening the webhook again.