Wrong test count in slack when using playwright-slack-reporter from Bitbucket pipeline

[Alona-T]

I do use standard setup for playwright-slack-reporter in my config

For some reason when i run tests from Bitbucket - i do have wrong test pass/fail/skip count in case i do have retries set

retries: process.env.CI ? 2 : 0,

If i do it from local - there are no issues

Is there a way to fix it when run on Bitbucket?

[ryanrosello-og]

hey @Alona-T ,
I would recommend having 2 playwright config files: 1 for your local and the other for Bitbucket, that way you wouldn't need to do any ternary logic in your config.

Can you try running the Playwright reporter by itself on Bitbucket and see what results you get?
And then try re-running it using just the normal html reporter and then see what results you get?

[Alona-T]

@ryanrosello-og
I tried to divide configs and added one for CI specifically using:
retries: 2,
workers: 1,

Also tried to run on CI same config with html report only - same wrong result

What i do find weird is - it does retry on passed tests as well (and i suspect it calculates these retries for some reason and sending that count to slack

So as far as i see failed test cases count is shown correctly in slack however passed and skipped - they are not. So maybe you guys can support it in your code where retries for same test case will be counted as 1 for all type of results?

[ryanrosello-og]

@Alona-T can you try one more thing. Would you be able to rerun using just a plain html reporter and take screenshot of the stats. Just want to compare things. Also can you let me know if you are using projects within your playwright config and what is the structure of your specs (e.g. are the tests deeply nested/dynamic tests?)

[Alona-T]

@ryanrosello-og i actually tried to run plain playwright-slack-report without any custom reports locally with 2 retries and it gave me wrong calculation for skipped and passed when there is a failure present (check point 2)

If there are no failures - it did give correct count (check point 1)

In terms of tests - i do have describe/test structure, nothing super complicated being honest

[miguelofbc]

Thanks for reporting @Alona-T ! I am experiencing the same @ryanrosello-og . The retries are messing up the counting for skipped.

[ryanrosello-og]

@Alona-T , struggling to reproduce this.

This is my playwright config:

import type { PlaywrightTestConfig } from "@playwright/test";
import { devices } from "@playwright/test";

const config: PlaywrightTestConfig = {
  testDir: "./tests",
  timeout: 30 * 1000,
  expect: {
    timeout: 5000,
  },
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: 2,
  workers: 1,
  reporter: [
    ["dot"],
    [
      "./node_modules/playwright-slack-report/dist/src/SlackReporter.js",
      {
        channels: ["chan-chan"],
        sendResults: "always", // "always" , "on-failure", "off",
        meta: [
          {
            key: "Branch",
            value: "feature/test-slack-report",
          },
          {
            key: "Build",
            value: "v1.08-RC_2331150.033",
          },
        ],
      },
    ],
    ["html"],
  ],
  use: {
    actionTimeout: 0,
    trace: "on-first-retry",
    screenshot: "only-on-failure",
    video: "retain-on-failure",
  },
  projects: [
    {
      name: "chromium",
      use: {
        ...devices["Desktop Chrome"],
      },
    },
  ],
};

export default config;

Can you tell me a little bit more about your project?

• Are you using projects in the playwright config?
• Are you using the page supplied by playwright or are you handling the creation of browsers yourself?
• Are you using serial mode within your tests?
• How many workers are you using?

[Alona-T]

@ryanrosello-og
Are you using projects in the playwright config?

• Not for this one as it is API testing (no browser needed)

Are you using the page supplied by playwright or are you handling the creation of browsers yourself?

• this particular error was in API project (no browsers involved)

Are you using serial mode within your tests?

• Yes we do use it
• Example:
  test.describe.serial('Test name', () => {} or

test.describe.configure({ mode: "serial" }); test.describe.serial(POST /execute for unauthed user @coins, () => {}

How many workers are you using?

• 1 worker

[ryanrosello-og]

I believe this will be fixed by #54

Please try v1.1.24 Release @miguelofbc, @Alona-T