ryanrosello-og/playwright-slack-report

CLI option doesn't parse merged shard json results

IvayloStoychev opened this issue · 4 comments

IvayloStoychev commented

I am using option C to send results to Slack. I started with example configuration like this:

{ "sendResults": "always", "slackLogLevel": "error", "sendUsingBot": { "channels": ["test_auto_slack_message"] }, "showInThread": true, "meta": [ { "key": "build", "value" : "1.0.0"}, { "key": "branch", "value" : "master"}, { "key": "commit", "value" : "1234567890"}, { "key": "results", "value" : "https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png"} ], "maxNumberOfFailures": 4, "disableUnfurl": true }

And also I have following configurations in yaml file:

  - name: Merge all blob reports
        if: always()
        run: npx playwright merge-reports --reporter json ./blob-report > e2e-results.json

      - name: Install slack-reporter
        if: always()
        run: npm i playwright-slack-report

      - name: View merged results
        run: cat ${GITHUB_WORKSPACE}/e2e-results.json
  
      - name: Send report to Slack using CLI
        env:
          SLACK_BOT_USER_OAUTH_TOKEN: ${{ secrets.SLACK_BOT_USER_OAUTH_TOKEN }}
        run: npx playwright-slack-report --config="${GITHUB_WORKSPACE}/cli_config.json" --json-results="${GITHUB_WORKSPACE}/e2e-results.json"

### **When the job is triggered I have received following error:** 

    `Run npx playwright-slack-report --config="${GITHUB_WORKSPACE}/cli_config.json" --json-results="${GITHUB_WORKSPACE}/e2e-results.json"
  npx playwright-slack-report --config="${GITHUB_WORKSPACE}/cli_config.json" --json-results="${GITHUB_WORKSPACE}/e2e-results.json"
  shell: /usr/bin/bash -e {0}
  env:
    CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /home/runner/work/storefront-e2e-tests/storefront-e2e-tests/gha-creds-f7c7467dcf86e7c5.json
    GOOGLE_APPLICATION_CREDENTIALS: /home/runner/work/storefront-e2e-tests/storefront-e2e-tests/gha-creds-f7c7467dcf86e7c5.json
    GOOGLE_GHA_CREDS_PATH: /home/runner/work/storefront-e2e-tests/storefront-e2e-tests/gha-creds-f7c7467dcf86e7c5.json
    CLOUDSDK_CORE_PROJECT: team-operations
    CLOUDSDK_PROJECT: team-operations
    GCLOUD_PROJECT: team-operations
    GCP_PROJECT: team-operations
    GOOGLE_CLOUD_PROJECT: team-operations
    CLOUDSDK_METRICS_ENVIRONMENT: github-actions-setup-gcloud
    CLOUDSDK_METRICS_ENVIRONMENT_VERSION: 1.1.1
    SLACK_BOT_USER_OAUTH_TOKEN: ***
##[debug]/usr/bin/bash -e /home/runner/work/_temp/dbb1a3db-39f2-491f-9797-b7f2216957df.sh
/home/runner/work/storefront-e2e-tests/storefront-e2e-tests/node_modules/playwright-slack-report/dist/src/ResultsParser.js:48
            passed: parsedData.stats.expected,
                                     ^

TypeError: Cannot read properties of undefined (reading 'expected')
    at ResultsParser.parseFromJsonFile (/home/runner/work/storefront-e2e-tests/storefront-e2e-tests/node_modules/playwright-slack-report/dist/src/ResultsParser.js:48:38)
    at async Command.<anonymous> (/home/runner/work/storefront-e2e-tests/storefront-e2e-tests/node_modules/playwright-slack-report/dist/cli.js:32:27)

Node.js v[18](https://github.com/xxxx/storefront-e2e-tests/actions/runs/7581732646/job/20650052808#step:17:18).[19](https://github.com/xxxx/storefront-e2e-tests/actions/runs/7581732646/job/20650052808#step:17:19).0
Error: Process completed with exit code 1.`

Where my e2e-results.json has following data:

{ "config": { "forbidOnly": false, "fullyParallel": false, "globalSetup": null, "globalTeardown": null, "globalTimeout": 0, "grep": {}, "grepInvert": null, "maxFailures": 0, "metadata": { "totalTime": 87598.[30](https://github.com/xxxx/storefront-e2e-tests/actions/runs/7581732646/job/20650052808#step:16:31)7, "actualWorkers": 2 }, "preserveOutput": "always", "projects": [], "reporter": [ [ "dot" ] ], "reportSlowTests": { "max": 5, "threshold": 15000 }, "rootDir": "/home/runner/work/storefront-e2e-tests/storefront-e2e-tests", "column": 6, "specs": [ { "title": "TBD-#: Wishlist is empty when the only product in default wishlist is removed @wip", "ok": true, "tags": [ "wip" ], "tests": [ { "timeout": 180000, "annotations": [], "expectedStatus": "passed", "projectId": "e2e-tests1eb99aebfb39e9e7", "projectName": "e2e-tests", "results": [ { "workerIndex": 0, "status": "passed", "duration": 11844, "errors": [], "stdout": [ { "text": "\u001b[[32](https://github.com/xxxx/storefront-e2e-tests/actions/runs/7581732646/job/20650052808#step:16:33)m[2024-01-19T09:16:57.589] [INFO] Data Setup - \u001b[39mSharding index is '2'\n" }, { "text": "\u001b[32m[2024-01-19T09:17:03.909] [INFO] Data Setup - \u001b[[39](https://github.com/xxxx/storefront-e2e-tests/actions/runs/7581732646/job/20650052808#step:16:40)mCreate new Admin storage session file '.auth/adminStorageState-2-0-19012024.json'\n" }, { "text": "\u001b[32m[2024-01-19T09:17:03.971] [INFO] Data Setup - \u001b[39mSharding index is '2'\n" }, { "text": "\u001b[32m[2024-01-19T09:17:03.971] [INFO] Data Setup - \u001b[39mStorefront storage session file '.auth/storefrontStorageState-2-0-19012024.json' already exists and will be used\n" } ], "stderr": [], "retry": 0, "startTime": "2024-01-19T09:16:57.576Z", "attachments": [] } ], "status": "expected" } ], "id": "44d9804bb6e7d24b5c78-e6a3be574a8a4a0f57501eb99aebfb3 [e2e-results.json](https://github.com/ryanrosello-og/playwright-slack-report/files/13987182/e2e-results.json) 9e9e7", "file": "storefronts/wishlist/default/wishlist.page.general.spec.ts", "line": 75, "column": 3 } ] } ] } ], "errors": [] }

ryanrosello-og commented

hey @IvayloStoychev , can you confirm what version of Playwright you are using?

stats.expected was only introduced in Playwright v1.40, if you are using a older version (older the 1.40) then its possible this CLI app wont work for you

IvayloStoychev commented

hey @ryanrosello-og thank you for the quick response. I was using Playwright 1.37, after update to last version 1.41 the issue is no longer reproducible and the report is sent to slack! Thank you!
However I have one more question related to the CLI slack reporter.
Is there any way to adjust the cli_config.json file to send a customised message like my previous message generated through the reported in playwright.config.ts file

Generated through CLI slack reporter:

image

Generated through playwright.config.ts slack reporter:

image

Like, how to add divider, how to send different mrkdwn lines in the message, can I bold some text, can I add some description for the passed/failed tests (like in the screenshot 2) . Or if there is a more detailed documentation about what and how can be configure in cli_config.json

ryanrosello-og commented

hi @IvayloStoychev unfortunately custom layout is not yet supported via the CLI, feel free to create a separate github issue. I'll try and figure out how to get it to work.

IvayloStoychev commented

@ryanrosello-og Thanks for the advice. I will create a separate issue for this topic as well