Playwright: Seems impossible to customize test.module

[nmilcoff]

test.module (for the module) is set here https://github.com/DataDog/dd-trace-js/blob/master/packages/dd-trace/src/plugins/util/test.js#L418

The function is called (in the case of playwright) in this line https://github.com/DataDog/dd-trace-js/blob/master/packages/dd-trace/src/plugins/ci_plugin.js#L70 using this.constructor.id.

Unfortunately there's no way (unless I'm missing something) for a consumer to customize the test module name.

Also tried doing this via an annotation (located in an automated fixture):

testInfo.annotations.push({
        type: 'DD_TAGS[test.module]',
        description: "my_module",
      })

And this way the test can have its module updated (checked in "Other tags"), but the suite or module don't seem to be updated ("playwright" remains as value if I inspect "Other tags" for the module/suite).

This limitation makes DataDog's Test Visibility view for Modules to always show a single module. In my use case, I have more than one playwright module and I'd like to be able to filter by these in DataDog's UI.

[juan-fernandez]

hi @nmilcoff ! Thanks for bringing this to our attention. This is indeed impossible at the moment, but it could be considered as a feature request.

For me to understand what you're trying to accomplish, what do you mean by "playwright module"? Correct me if I'm wrong, but as far as I know, there's no standard "module" concept in playwright.

[nmilcoff]

For me to understand what you're trying to accomplish, what do you mean by "playwright module"? Correct me if I'm wrong, but as far as I know, there's no standard "module" concept in playwright.

That is correct. In my use case, I have several predefined test runs/collections of tests (I use tags to select which test I want to run on each case). These predefined test runs would run at different times/under different circumstances (e.g. a daily run, a regression run, a healthcheck run, ...).
I would like to use DataDog's test.module as a filter for these in Test Visibility (one could think of these predefined test runs as test modules I suppose). The alternative is to use a custom tag, but then the majority of the predefined views (that have a fixed filter for session/module/suite/test) would have very little value for me.

[juan-fernandez]

For me to understand what you're trying to accomplish, what do you mean by "playwright module"? Correct me if I'm wrong, but as far as I know, there's no standard "module" concept in playwright.

That is correct. In my use case, I have several predefined test runs/collections of tests (I use tags to select which test I want to run on each case). These predefined test runs would run at different times/under different circumstances (e.g. a daily run, a regression run, a healthcheck run, ...). I would like to use DataDog's test.module as a filter for these in Test Visibility (one could think of these predefined test runs as test modules I suppose). The alternative is to use a custom tag, but then the majority of the predefined views (that have a fixed filter for session/module/suite/test) would have very little value for me.

Thanks for the context!

Do you run the tests associated to a given tag in different commands?

# job 1
npx playwright test --grep @daily
# job 2
npx playwright test --grep @regression
# job 3
npx playwright test --grep @healthcheck

if that's the case, could you use the @test.command tag to distinguish executions of different playwright tags? A command running tests is mapped to a "test session". I wonder if you could use the test session aggregation instead

[nmilcoff]

Do you run the tests associated to a given tag in different commands?
Yes, however these predefined test runs would typically comprehend more than a single session in the CI agent (mostly for performance reasons).

So my scripts in CI look more like this:
npx playwright test --grep @daily.app-domain1
npx playwright test --grep @daily.app-domain2

if that's the case, could you use the @test.command tag to distinguish executions of different playwright tags? A command running tests is mapped to a "test session". I wonder if you could use the test session aggregation instead

I think so, thank you for that suggestion! However I'd still need to add a custom tag to group multiple sessions into one, to match my definition of a predefined run, right?
If so, I think this issue is still relevant because if the session and module tags/definitions could be customized, I wouldn't have to use a custom tag and I would be able to use this default filter in DD's UI:

[juan-fernandez]

thanks for the extra context!

I think so, thank you for that suggestion! However I'd still need to add a custom tag to group multiple sessions into one, to match my definition of a predefined run, right?

indeed you would. In our platform, modules are a "smaller" aggregation than sessions though, so they can't be used for your purpose, if I understood you correctly. We could do a better job at explaining the nuances, for sure.

What might work for you is to use a different Test Service for each group:

DD_SERVICE=daily-tests npx playwright test --grep @daily.app-domain1
DD_SERVICE=daily-tests npx playwright test --grep @daily.app-domain2
...
DD_SERVICE=regression-tests npx playwright test --grep @regression.app-domain1
DD_SERVICE=regression-tests npx playwright test --grep @regression.app-domain2

This would separate executions from each type in the dedicated test service pages. I'm not sure if this would work for you but it might be worth trying. The disadvantage is that these would show up as "independent" in our product, but we're working in a repository-centric view which might help with that.

[nmilcoff]

@juan-fernandez thank you! Yeah that makes sense. It's just not ideal that my Test Services will not match my real services, but I guess that's not a big deal

[juan-fernandez]

thanks for the follow up, @nmilcoff. I'll close this because we don't really want to allow modifying test.module right now. If you have other ideas or feedback to improve the test service setup, feel free to reopen or create another issue 😄