Exception in root level after() doesn't fail test run in interactive mode

Question

Exception in root level after() doesn't fail test run in interactive mode

amitzur opened this issue 6 years ago · 16 comments

Current behavior:

If there's an after() outside of any describe, it or context ("rool level"), which throws an exception, that doesn't fail the test run. This is different from how it works in mocha, where you get a "after all" hook failure.

Also, if it's a synchronous failure, the exception doesn't show up anywhere.
If it's async, it shows up like this:

Desired behavior:

The whole test run should fail (sometimes you would want to place an assertion there that is possible only after all tests have run).

Steps to reproduce:

Add this to your tests:

after(() => {
  cy.then(() => {
    throw new Error('bla bla');
  });
});

Versions

Cypress: 3.0.3
OS: Mac OS X 10.12.6 (Sierra)
Browser: both Electron (59) and Chrome (68)

Answer 1 · 2018-08-08T08:40:17.000Z

I just found out that running cypress run or through the module API does report a failed test. It's just cypress open that shows the test as green.

Answer 2 · 2018-08-08T21:31:07.000Z

Yes, this is likely a UI level issue if that is the case.

Answer 3 · 2018-08-16T09:43:35.000Z

Hi there, is there progress in investigation of this issue?

Answer 4 · 2018-08-16T12:18:12.000Z

@amitzur considering this is only a UI bug, it's a pretty low priority at the moment, but PRs are welcome!

Answer 5 · 2018-08-16T12:28:34.000Z

glad to take the challenge. Any chance to get a head start on code? I already got familiar with it, but where are test results communicated to the client?

Answer 6 · 2018-08-16T14:09:00.000Z

@amitzur The code for this is in our reporter package which is a React app. I'm not totally sure where this discrepancy is at - but here's a link to the runnable area of the app (may not be where the error is), but hopefully gets you started. https://github.com/cypress-io/cypress/blob/develop/packages/reporter/src/runnables/runnable-and-suite.jsx#L8

Answer 7 · 2018-08-16T15:22:59.000Z

great, thanks!
2 things:

The URL you listed is from the whitelist branch, it needs to be fixed to master or develop.
How do you setup React devtools? Via a browser extension or a standalone app? I couldn't easily find how to inject <script src="http://localhost:8097"></script> if it's a standalone app, or how to provide a path to the extension via a plugin.

Answer 8 · 2018-08-16T15:39:00.000Z

Oops, yeah, should be PR from develop
@chrisbreiding

Answer 9 · 2018-08-16T15:55:44.000Z

If you run in Chrome, you should be able to install the extension the standard way by visiting https://chrome.google.com/webstore/detail/react-developer-tools/fmkadmapgofadopljbjfkapdkoienihi

Answer 10 · 2019-05-02T09:01:48.000Z

Example repo of failure: https://github.com/danielschwartz85/after-error-not-shown provided in #3348

Answer 11 · 2019-05-02T10:43:13.000Z

Trying to find the root of this issue:

It looks like the test:after:run event is never fired after the after runs and throws, although this test:after:run event is fired when there is an error thrown in the other hooks (before, etc).

Because of this, my suspicion is that the runnables object is not being built properly, so the Test Runner is not displaying the failure properly.

I tested that if the correct runnables object is sent to the UI, that it's rendered properly, so I think the object must be malformed somewhere.

afterEach failure runnable

{
  "id": "r3",
  "title": "test here",
  "hookName": "after each",
  "err": {
    "message": "bla\n\nBecause this error occurred during a 'after each' hook we are skipping all of the remaining tests.",
    "name": "Error",
    "stack": "Error: bla\n\nBecause this error occurred during a 'after each' hook we are skipping all of the remaining tests.\n    at Context.<anonymous> (http://localhost:56948/__cypress/tests?p=cypress/integration/issue_674_spec.coffee-840:4:11)"
  },
  "state": "failed",
  "failedFromHookId": "h1",
  "body": "function() {\n    return cy.window();\n  }",
  "type": "test",
  "duration": 328,
  "wallClockStartedAt": "2019-05-02T10:13:47.366Z",
  "wallClockDuration": 409,
  "timings": {
    "lifecycle": 47,
    "test": {
      "fnDuration": 22,
      "afterFnDuration": 1
    },
    "after each": [
      {
        "hookId": "h1",
        "fnDuration": 22,
        "afterFnDuration": 306
      }
    ]
  }
}

after failure runnable

{
  "id": "r3",
  "title": "test here",
  "state": "passed",
  "body": "function() {\n    return cy.window();\n  }",
  "type": "test",
  "duration": 75,
  "wallClockStartedAt": "2019-05-02T10:15:46.442Z",
  "wallClockDuration": 82,
  "timings": {
    "lifecycle": 45,
    "test": {
      "fnDuration": 28,
      "afterFnDuration": 2
    }
  }
}

Answer 12 · 2019-10-23T08:34:12.000Z

Hi, any updates / plans regarding when this would be fixed ?
Thanks.

Answer 13 · 2020-04-23T14:38:28.000Z

The code for this is done in cypress-io/cypress#7035, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

Answer 14 · 2020-04-28T05:59:29.000Z

Sorry, we accidentally merged the PR above prematurely. This is still in the review stage. We will close this issue and comment when this issue is merged for the next upcoming release.

Answer 15 · 2020-05-28T15:18:20.000Z

The code for this is done in cypress-io/cypress#7154, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

Answer 16 · 2020-06-08T20:34:17.000Z

Released in 4.8.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v4.8.0, please open a new issue.