Code-Coverage plugin does not work reliably when run in an Azure Pipeline

[lukitroe]

Logs and screenshots

Cannot find coverage file C:\azure_devops_agent\_work\8\s\packages\cypress-honeybee-e2e\.nyc_output\out.json
Skipping coverage report

Versions

• What is this plugin's version?
  • "@cypress/code-coverage": "^3.10.4"
• What is Cypress version?
  • "cypress": "^12.5.0"
• What is your operating system?
  • Tried on Windows 10 and Ubuntu Linux (20.04), both OSs show the same behavior
• What is the shell?
  • Windows: tried with CMD and Powershell Core
  • Linux: tried with BASH and Powershell Core
• What is the Node version?
  • node 18.15.0
• What is the NPM version?
  • npm 8.19.3
• How do you instrument your application?
  • The MFEs are instrumented with Babel and the Instanbul plugin
• When running tests, if you open the web application in regular browser, and open DevTools, do you see window.__coverage__ object? Can you paste a screenshot?
  • I had to blur the picture due to confidential information in it but I hope you get the impression of the window.__coverage__ object
  • 
• Is there .nyc_output folder? Is there .nyc_output/out.json file. Is it empty? Can you paste at least part of it so we can see the keys and file paths?
  • Here is the basic problem. Please see the bug description below.
• Do you have any custom NYC settings in package.json (nyc object) or in other NYC config files
  • package.json:

    "nyc": {
        "extends": "@istanbuljs/nyc-config-typescript",
        "all": true,
        "exclude": [ /* a list of glob patterns */ ] 
     } 

• Do you run Cypress tests in a Docker container?
  • No

Describe the bug
If we ran the pipeline e.g. three times consecutively, there is the possibility to have two times a .nyc_output folder and an out.json file with data in it and one time we will have an unsuccessful attempt (with the error message above). But there is also the possibility to have it the other way around (one successful and two unsuccessful), three successful attempts, or no successful attempt at all. No changes are made to the test code or the NYC configuration between the runs.
Additionally, I tried to create a .nyc_output/out.json file with an empty JSON object (e.g. { } before the runs, which lead to another error message and no data in out.json. So I removed this intermediate step again.

[marktnoonan]

Thanks for reporting this @lukitroe and apologies for the slow response.

I'm going to attempt to reproduce, or at least speculate as to some possible causes for this. While I look into it from that side could you confirm:

• This is still the same today in the latest version of Cypress (12.16.0)
• You don't experience this problem outside of an Azure pipeline - can't reproduce locally?

Additionally, I tried to create a .nyc_output/out.json file with an empty JSON object (e.g. { } before the runs, which lead to another error message and no data in out.json

This was a reasonable thing to try, sorry you are experiencing this. Intermittent CI failures are the worst.

[marktnoonan]

This issue is possibly related, although what is reported there doesn't seem to be intermittent #361