[Feature] Combine sharding with html reporter

Question

[Feature] Combine sharding with html reporter

bamanczak opened this issue 3 years ago · 50 comments

Currently there is no way to combine test results into one html report, while using sharding on multiple machines. (v1.17.0-rc1)

I end up with multiple reports which makes it harder for sharing with others in case of failures.

It would be nice to be able to combine multiple executions into 1 report.

I believe anyone who wants to scale tests horizontally might benefit from such feature

Answer 1 · 2021-11-19T17:14:46.000Z

Sounds reasonable to me! What kind of api would you like to combine the reports together?

Answer 2 · 2021-11-22T09:49:14.000Z

I personally have no preferences for how to merge the reports together. Whatever I can do on CI in a fast way.

Answer 3 · 2022-03-03T12:15:31.000Z

Since I'm currently using https://plugins.jenkins.io/allure-jenkins-plugin/ to merge multiple Allure reports and it is as simple as 👇 then I'd favor a similar approach to HTML reporter but any other out-of-the-box no-brainer solution would work as well.

            script {
                allure([
                        commandline      : 'allure-2.x',
                        includeProperties: false,
                        jdk              : '',
                        results          : [
                                [path: 'allure-results1'],
                                [path: 'allure-results2']
                        ]
                ])

Answer 4 · 2022-03-21T17:19:22.000Z

This feature would be great to have. We are looking to migrate our E2E testing solution to Playwright. We have a component library where each test runs several variations of a particular test to check for visual regressions. Some of the variants we have are:

LTR vs RTL modes
Desktop vs mobile viewports
Different themes for the components

Right now the results for these tests can get split up across multiple shards. It would be great to have a single result where we can see all of the test results.

Answer 5 · 2022-03-28T17:23:26.000Z

I was able to build one solution for my use case (in Node.js). I had multiple reports from different shards (tests distributed with a custom logic) in a single run and wanted to send a single merged html report. Hope it helps.

Installation

npm install playwright-merge-html-reports --dev

Usage

const { mergeHTMLReports } = require("playwright-merge-html-reports");

mergeHTMLReports([
  process.cwd() + "/html_report-1",
  process.cwd() + "/html_report-2"
])

It creates a separate HTML merged report, folder name - merged-html-report (customizable)

⚠️ Tested on some use cases: Still at early stages of development

Github Repo
NPM package

Answer 6 · 2022-03-28T19:59:33.000Z

@anooprav7 thanks kind men. I will try this in a week or two.

Answer 7 · 2022-04-28T06:43:12.000Z

Is this feature still in progress?

Answer 8 · 2022-05-16T16:01:44.000Z

My test suite becomes bigger and bigger every week, and this is a blocker for my team to start using sharding.

The solution mentioned by anooprav7 seems cool, but we would like to have a stable version of such toolkit. I was really hoping for it to appear in 1.22 :(

Answer 9 · 2022-06-02T02:19:34.000Z

We're running our tests on GitHub Actions. We'd ideally like a single report, but for now will create a report per shard/matrix runner.

I'm not sure how this should work - there's no easy mechanism for the runners to talk to each other to combine a report.

Answer 10 · 2022-06-10T06:02:18.000Z

wanting this too. We run our tests back to back with various different configurations and want to combine the final test results into a single html. So pls team at MS and Playwright, at some point in future, make this a native playwright feature.
In the meantime, we have wired in the module provided by @anooprav7 . There are a bunch of issues with that module that we needed to address so just a heads up to anyone else try it.

We use NodeJS 12.X. This limits us to Playwright 1.21.X or lower. The module looks for fs/promises but cant find it. We've had to alias in the fs.promises module to that path.
If a test is successful and no data is generated as a result (in the 'data' directory) it also breaks, as it cant find the dir. So we've mitigated it by manually creating that dir no matter what.
I believe there is a PR to fix both the above waiting to merged in gitHub for that module. But as for now there is no ETA. @anooprav7 pls let us know when the fix is available.

Answer 11 · 2022-06-11T09:12:04.000Z

Feel free to install from my (unpublished) fork in the meantime, @mstepin. We're using this on one of our repositories at Vercel right now and it seems like we have everything working.
https://github.com/mrmckeb/playwright-merge-html-reports/tree/with-dist

npm i https://github.com/mrmckeb/playwright-merge-html-reports#aadf6a7

Answer 12 · 2022-06-11T10:20:37.000Z

The fix has been merged and published - playwright-merge-html-reports@0.2.2. Thank you @mrmckeb and @nickofthyme for the fixes (anooprav7/playwright-merge-html-reports#2 and anooprav7/playwright-merge-html-reports#3).

Answer 13 · 2022-07-01T07:02:46.000Z

Looking forward to this feature.
Our tests take more than 1 hour to finish using one Jenkins job.
We have to split the tests into mutiple Jenkins jobs to run across mutiple Jenkins agents using Playwright sharding feature.
However, getting a single HTML test report is an issue with this solution. We need to pass HTML test report for release team to review before they do release.

Answer 14 · 2022-07-01T14:16:46.000Z

@TimeInvestor just to be clear, you can currently shard your playwright tests across multiple machines and use playwright-merge-html-reports@0.2.3 npm package to combine all the html reports into one.

Answer 15 · 2022-07-01T14:37:55.000Z

@TimeInvestor just to be clear, you can currently shard your playwright tests across multiple machines and use playwright-merge-html-reports@0.2.3 npm package to combine all the html reports into one.

Thanks @nickofthyme. Will check it out although I am not sure if it works well for my scenario that the test results are residing with multiple Jenkins jobs over different Jenkins slaves.

Answer 16 · 2022-07-01T14:55:42.000Z

Yeah I have it running in parallel and merging all reports across 10 agents in buildkite. The only trick is to run all test with the --shard=n/x option then after each job completes, store the report in an archive or otherwise. Then have a job that depends on the parallelized jobs that runs only after all jobs are complete across all the agents. Then collect all the reports from all the parallel jobs.

Another difficultly is deciding how to build, run or host your application for testing. I decided on just building our react app first then pulling down the files in each parallel job and running a local webServer.

It takes a bit to get it just right but the whole file archive processes and terminology is different on every ci provider.

Hope that helps, but if you run into issues with reports merging with playwright-merge-html-reports please open an issue in the GH repo and I'll take a look.

Answer 17 · 2022-07-28T05:46:32.000Z

@JoelEinbinder I was chatting with my colleagues about this today and wanted to share thoughts.

What we're doing now is running multiple shards (GitHub Actions), uploading each HTML report as an artifact, and then we have a final step that combines all the uploaded reports using playwright-merge-html-reports.

How we think an official solution might look would be that there's a new reporter which outputs the information required by the HTML reporter (perhaps the raw-reporter), and again that would be uploaded/downloaded for processing. The HTML reporter would then process those into a single HTML report.

Considerations:

Uploaded artifacts will add up over time, so users will probably want to delete the shard reports once they've combined them.
If someone has to re-run a single shard, we'd ideally want a way to update the combined report with that shard's updated run without the other shard reports.

Answer 18 · 2022-09-26T15:28:36.000Z

What we're doing now is running multiple shards (GitHub Actions), uploading each HTML report as an artifact, and then we have a final step that combines all the uploaded reports using playwright-merge-html-reports.

I'm looking to do the same, but I'm a little confused as to how this works with each shard outputting an html report with the same name/filepath. Is each artifact uploaded with a different name? Thanks!

Answer 19 · 2022-09-26T16:07:16.000Z

I'm looking to do the same, but I'm a little confused as to how this works with each shard outputting an html report with the same name/filepath. Is each artifact uploaded with a different name? Thanks!

@angelo-loria Yes, exactly

Ideally you override the report output directory using PLAYWRIGHT_HTML_REPORT environment variable when you run the playwright test. This can be indexed by the shard number (i.e. ./html-reports/report-${n}). Then you could use node to merge all the available reports.

const fs = require('fs');
const { mergeHTMLReports } = require('playwright-merge-html-reports');

// get all archived/uploaded report directories locally into ./html-reports/report-*
getAllReports();

// find all directories in ./html-reports/*
const reportPathsToMerge = fs
    .readdirSync('./html-reports', { withFileTypes: true })
    .filter((item) => item.isDirectory())
    .map(({ name }) => path.resolve('html-reports', name));

mergeHTMLReports(reportPathsToMerge)

The report directory name is erroneous and does not impact the final report in any way, all the data used to merge comes only from the report content.

Another approach would be to run playwright test normally without an indexed output path. Then use some compression library like targz that allows you to set the indexed name when compressing and decompressing. This way you also save a little time when pulling down all the reports, with the reduced size.

At the end of the day the idea is simple but the implementation is highly dependent on the CI you are using and how all the jobs and steps play nicely together. It took a lot of trial and error for me to get it just right.

Answer 20 · 2022-09-26T21:01:20.000Z

Here's what I ended up with as far as an Actions workflow (sanitized) for this. It might end up being too convoluted for us to use in production and the merging somewhat negates the performance benefits of sharding, but it does work as expected and hopefully it'll help someone out. Thanks to everyone in this thread for their help.

jobs:
  playwright_test:
    name: Playwright Test
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3]
    defaults:
      run:
        working-directory: ./playwright
    steps:
      - uses: actions/checkout@v2

      - name: Setup NodeJS SDK ${{ env.NODEJS_VERSION }}
        uses: actions/setup-node@v3.1.1
        id: setup-node
        with:
          node-version: ${{ env.NODEJS_VERSION }}

      - name: Yarn Install
        run: yarn install

      - name: Setup
        run: yarn setup

      - name: Run Tests
        id: test
        run: yarn test --shard=${{ matrix.shard }}/${{ strategy.job-total }}

      - name: Upload Artifact
        uses: actions/upload-artifact@v2
        if: always()
        with:
          name: report-${{ matrix.shard }}
          path: ./playwright/playwright-report

  publish_pages:
    name: Publish to GitHub Pages
    if: always()
    needs: playwright_test
    runs-on: ubuntu-latest
    # needed for gh pages deploy
    environment:
      name: ${{ inputs.environment }}-report
      url: ${{ steps.deployment.outputs.page_url }}
    defaults:
      run:
        working-directory: ./playwright
    steps:
      - uses: actions/checkout@v2

      - name: Setup NodeJS SDK ${{ env.NODEJS_VERSION }}
        uses: actions/setup-node@v3.1.1
        id: setup-node
        with:
          node-version: ${{ env.NODEJS_VERSION }}
      
      - uses: actions/download-artifact@v2
        with:
          path: ./playwright/playwright-report

      - name: Yarn Install
        run: yarn install

      // this is a simple "node mergeReports.js" npm script
      - name: Merge Reports
        run: yarn merge

      - name: Upload Pages Artifact
        uses: actions/upload-pages-artifact@v1
        with:
          path: './playwright/html-report'

      - name: Setup Pages
        uses: actions/configure-pages@v1

      - uses: actions/download-artifact@v2
        with:
          name: 'github-pages'
  
      - name: Deploy to GitHub Pages
        id: pages-deploy
        uses: actions/deploy-pages@v1

      - name: Message PR of Artifact Published
        if: ${{ always() && github.event_name == 'pull_request' }}
        uses: thollander/actions-comment-pull-request@main
        with:
          message: |
            See Playwright Test Report at ${{steps.pages-deploy.outputs.page_url}}
          comment_includes: "See Playwright Test Report "
          pr_number: ${{ inputs.pr_number }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

My mergeReports.js:

const fs = require("fs");
const path = require("path");
const { mergeHTMLReports } = require("playwright-merge-html-reports");

const reportPathsToMerge = fs
    .readdirSync(process.cwd() + "/playwright-report", { withFileTypes: true })
    .filter((item) => item.isDirectory())
    .map(({ name }) => path.resolve(process.cwd() + "/playwright-report", name));

mergeHTMLReports(reportPathsToMerge, {
    outputFolderName: "html-report",
});

Answer 21 · 2022-11-07T08:46:39.000Z

Can it support Python?

Answer 22 · 2023-01-25T21:28:33.000Z

Is this feature still being considered? Im doing a POC on Playwright at work and this feature would be brilliant for us

Answer 23 · 2023-03-12T14:18:54.000Z

@mxschmitt I believe this was addressed?

And was reverted as far as I can see.

Answer 24 · 2023-03-28T13:14:08.000Z

The library that a lot of us are using no longer works as of the latest Playwright version 1.32

Rippling/playwright-merge-html-reports#2

It would be great to see if theres any progress with this native support of merging reports

Answer 25 · 2023-03-29T14:49:14.000Z

@costa-collibra - Yes native is the best solution, but have you tried using playwright-merge-html-report on v1.32?

Answer 26 · 2023-03-29T15:13:20.000Z

@nickofthyme yes I am using that library but the latest playwright version caused it to break

someone fixed it here - https://www.npmjs.com/package/@nil1511/playwright-merge-html-reports

this has fixed our builds

Answer 27 · 2023-03-29T15:18:14.000Z

Gotcha, sorry I was unaware they moved and renamed the repo & npm package.

Answer 28 · 2023-03-29T16:03:28.000Z

@nickofthyme i think someone needed it urgently so forked it, id expect this fix to go in the original repo too

Answer 29 · 2023-03-29T17:03:18.000Z

That's what I thought at first too, but the original repo points to the new location as the Rippling repo. Anyways... 😅

Answer 30 · 2023-04-03T15:41:33.000Z

one more option to merge shard reports

Answer 31 · 2023-04-24T18:16:36.000Z

I know blob report merging is coming soon which solves the sharded reports but I have a use case for merging full reports from independent test runs. @anooprav7 I create a PR with a CLI wrapper

Answer 32 · 2023-07-24T20:42:43.000Z

I came back to this issue today to check on any progress and I noticed the blob reporter merged back in April in v1.35.1. There was no doc on it so I wasn't sure if if the feature fully addressed this issue. For me at least, this turned out to be pretty great.

I enabled the blob reporter then ran:

npx playwright test --shard=1/3 
npx playwright test --shard=2/3 
npx playwright test --shard=3/3

All results were placed in the blob-report directory

then I ran
npx playwright merge-reports blob-report --reporter=html
and it produced a playwright-report directory with combined results.

and I then ran

npx playwright merge-reports blob-report --reporter=junit > playwright-report/results.xml

and it produced the xml output.
I guess the junit reporter must default standard out. I didn't expect to have to redirect the output but I guess it might not be reading the reporting configuration yet.
Overall, I'm pretty excited to have this functionality built in. Are there additional pieces coming that explain why this feature request is still open? Just trying to gauge the risks of using this before it's officially documented.

Answer 33 · 2023-07-24T20:57:15.000Z

It would be pretty cool if it produced a merged report for every report type that was already defined in the report config. If I left off the--reporter flag it didn't produce any reports and it didn't seem to fail either. Apologies if this is a work in progress. I am pretty psyched to have what's already there!

Answer 34 · 2023-07-25T09:43:06.000Z

@damongabrielle oh wow, i just replicated your steps and got it working against 3 shards.

Im a little reluctant to use it though without any official support from Playwright. This is pretty awesome though

Answer 35 · 2023-07-25T11:59:20.000Z

@damongabrielle @costa-collibra, yeah, it's working 😮, but I'm facing a problem when running on CI. The HTML report opens automatically if some of the tests fail. Unfortunately, merge-reports is ignoring the 'open' property in the Playwright config, and I don't see any other way to configure that. 😢 I'm not sure if I'm missing something, if there's a workaround for it, or maybe that's the reason why merging multiple reports is not officially supported yet.

There is even a documentation https://playwright.dev/docs/next/test-merge-report (it's marked as 'Version: Next' though)

Answer 36 · 2023-07-25T12:14:19.000Z

@damongabrielle @costa-collibra, yeah, it's working 😮, but I'm facing a problem when running on CI. The HTML report opens automatically if some of the tests fail. Unfortunately, merge-reports is ignoring the 'open' property in the Playwright config, and I don't see any other way to configure that. 😢 I'm not sure if I'm missing something, if there's a workaround for it, or maybe that's the reason why merging multiple reports is not officially supported yet.

There is even a documentation https://playwright.dev/docs/next/test-merge-report (it's marked as 'Version: Next' though)

try setting env variable PW_TEST_HTML_REPORT_OPEN: 'never'

Answer 37 · 2023-07-25T12:24:59.000Z

@MindaugasMateika Oh, now I see where I have made a mistake (I forgot to pass CI=true when running on my test setup). But setting it as an environment variable PW_TEST_HTML_REPORT_OPEN='never' works too! Thank you.

Answer 38 · 2023-07-25T13:00:06.000Z

I have noticed one problem with merging when using the setup test (https://playwright.dev/docs/test-global-setup-teardown#setup-example).

When running with --shard=1/2 and shard=2/2, the setup test is executed once per each shard (so 2x in my example). When merging these two tests, the resulting report is broken for these two tests.

When I click on these two tests, most of the resulting page is empty.

Other than that, it looks great!

Answer 39 · 2023-07-25T13:20:54.000Z

@swetrify is is running the global setup twice or just reporting on it twice?

If its running setup twice that seems a pretty big bug/oversight

Answer 40 · 2023-07-25T13:22:03.000Z

@damongabrielle @MindaugasMateika

I have this set

const reportList: ReporterDescription[] = [['list'], ['blob'], ['html', { open: 'never' }]];

to prevent the report from opening

Answer 41 · 2023-07-25T14:00:51.000Z

@swetrify is is running the global setup twice or just reporting on it twice?

If its running setup twice that seems a pretty big bug/oversight

The setup is running on each shard once, resulting in the global setup being executed twice (in my example with 2 shards). When I don't merge the reports, each HTML file contains a result of the global setup that differs from each other. I am not sure if it's a bug or a feature. The documentation for sharding and global setup is not clear on this matter.

Edit: I think it's a feature. It does make sense in the context of the login example. You need to create storageState on each shard; otherwise, one of the shards wouldn't be able to log in because it would be missing storageState.

Answer 42 · 2023-07-25T14:25:47.000Z

I'm surprised @costa-collibra that the report config open: 'never' is working for you during the execution of merge-reports. I have open: 'never' set but that doesn't work for me on merge-reports. That has always successfully prevented the report from opening at the end of a test run though. I wonder if you have additionally set CI=true or PW_TEST_HTML_REPORT_OPEN='never'. Both of those fixed it for me.

Answer 43 · 2023-07-25T14:29:41.000Z

@damongabrielle actually you are correct on that part I misread. The report does open when I merged. Its great though that you have a solution as we will need to set this in the pipeline.

Answer 44 · 2023-08-10T19:59:24.000Z

Here's another example of an Actions file using the blob reporter and merging CLI. This merges to XML for use with dorny-test-reporter and deploys the HTML report to GH Pages:

.
.
.
jobs:
  playwright_test:
    name: Playwright Test
    strategy:
      fail-fast: false
      matrix:
        shard: [1/3, 2/3, 3/3]
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: ./playwright
    steps:
      - uses: actions/checkout@v3

      - name: Setup NodeJS SDK ${{ env.NODEJS_VERSION }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODEJS_VERSION }}

      - name: Yarn Install
        run: yarn install
        
      - name: Setup
        run: yarn setup

      - name: Run Tests
        run: yarn test:ci --shard ${{ matrix.shard }}

      - name: Upload blob report to GitHub Actions Artifacts
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: all-blob-reports
          path: ./playwright/blob-report/
          retention-days: 10
      
      # screenshot/video/trace is stored in test-results if a test fails, otherwise empty if all tests pass
      - name: Upload test-results to GitHub Actions Artifacts
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: test-results
          path: ./playwright/test-results/
          retention-days: 10
          if-no-files-found: warn
          
  reporting:
    if: always()
    needs: [playwright_test]
    name: Reporting
    outputs:
      page_url: ${{ steps.deployment.outputs.page_url }}
    environment:
      name: ${{ inputs.environment }}-report
      url: ${{ steps.deployment.outputs.page_url }}
    defaults:
      run:
        working-directory: ./playwright

    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
 
    - uses: actions/setup-node@v3
 
    - name: Yarn Install
      run: yarn install

    - name: Download blob reports from GitHub Actions Artifacts
      uses: actions/download-artifact@v3
      with:
        name: all-blob-reports
        path: ./playwright/all-blob-reports
    
    # test-results directory is only created when a test fails, if all pass there is no dir to download
    - name: Check for test-results Artifact
      uses: xSAVIKx/artifact-exists-action@v0
      id: check-artifact
      with:
        name: test-results

    # TODO - watch for upcoming ignore feature on this so that the above step can be removed
    - name: Download test-results from GitHub Actions Artifacts
      uses: actions/download-artifact@v3
      if: ${{ steps.check-artifact.outputs.exists == 'true' }}
      with:
        name: test-results
        path: ./playwright/test-results
      
    # xml report use by dorny-test-reporter
    - name: Merge into XML Report
      run: PLAYWRIGHT_JUNIT_OUTPUT_NAME=junit.xml npx playwright merge-reports --reporter=junit ./all-blob-reports
        
    - name: Test Report
      uses: dorny/test-reporter@v1
      if: success() || failure()
      with:
        name: Playwright Test Report
        path: "**/junit.xml"
        reporter: java-junit
        fail-on-error: 'false'

    - name: Merge into HTML Report
      run: npx playwright merge-reports --reporter html ./all-blob-reports 

    - name: Upload HTML report
      uses: actions/upload-pages-artifact@v1
      with:
        name: github-pages
        path: './playwright/all-blob-reports'
        retention-days: 14
        
    - name: Setup Pages
      uses: actions/configure-pages@v3

    - uses: actions/download-artifact@v3
      with:
        name: github-pages

    - name: Deploy to GitHub Pages
      id: deployment
      uses: actions/deploy-pages@v1

Answer 45 · 2023-08-11T10:29:37.000Z

Hey everybody,

This has been released in 1.37.

Documentation: https://playwright.dev/docs/test-sharding
Release notes: https://github.com/microsoft/playwright/releases/tag/v1.37.0
Video overview: https://youtu.be/cEd4SH_Xf5U

Thank you all for the feedback!

Answer 46 · 2023-08-12T12:55:33.000Z

Hi, does this also support mergin of custom reports? For example we use a custom json reporter.
If not do you have any hints on how we could accomplish this?

I assume I can implement a merge method in my reporter?

Answer 47 · 2023-08-14T12:00:24.000Z

@marwie custom reports are supported as well. Make sure to specify it when using the merge-repots CLI tool and it should all just work.

Answer 48 · 2023-08-14T12:21:40.000Z

Hi @aslushnikov thanks for the reply.

Should i pass in the path to my reporter as an argument then and is it possible to specify multiple reporters (e.g. can I both use the html and my reporter like so: --reporter html ./myreporter.js ?)

Answer 49 · 2023-08-14T13:36:14.000Z

@marwie you'd need to:

create a custom config for the merge-reports tool
configure your custom reporter in that config: https://playwright.dev/docs/next/test-reporters#custom-reporters
use the --config option to the merge-reports tool to pass in the config: https://playwright.dev/docs/next/test-sharding#merge-reports-cli

Answer 50 · 2024-03-04T06:10:00.000Z

Hi,
I have a requirement where I have to combine the reports for that I am using playwright-merge-html-reports
@anooprav7
This is generating the reports with html content only, I also want to merge the xml report with that, do we have a way to do this via this library