Support mocha test retries / retry test failures
cameronc56 opened this issue · 77 comments
Is this a Feature or Bug?
Feature
Current behavior:
With this.retries set, when a test fails:
TypeError: Cannot set property 'err' of undefined
at Reporter.mergeErr (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/reporter.js:95:18)
at Reporter.parseArgs (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/reporter.js:199:20)
at Reporter.emit (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/reporter.js:190:23)
at Object.server.startWebsockets.onMocha (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/project.js:287:22)
at Socket.<anonymous> (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/socket.js:237:36)
at emitThree (events.js:116:13)
at Socket.emit (events.js:194:7)
at /home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/socket/node_modules/socket.io/lib/socket.js:503:12
at _combinedTickCallback (internal/process/next_tick.js:67:7)
at process._tickCallback (internal/process/next_tick.js:98:9)
TypeError: Cannot set property 'err' of undefined
at Reporter.mergeErr (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/reporter.js:95:18)
at Reporter.parseArgs (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/reporter.js:199:20)
at Reporter.emit (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/reporter.js:190:23)
at Object.server.startWebsockets.onMocha (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/project.js:287:22)
at Socket.<anonymous> (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/socket.js:237:36)
at emitThree (events.js:116:13)
at Socket.emit (events.js:194:7)
at /home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/socket/node_modules/socket.io/lib/socket.js:503:12
at _combinedTickCallback (internal/process/next_tick.js:67:7)
at process._tickCallback (internal/process/next_tick.js:98:9)
Desired behavior:
this.retries is supported at the describe() and it() level
We've internally talked about this before and we'll be doing something different from the way mocha implements retries.
It will work the same - but will be controlled by different API's than what mocha provides. I'll open a proposal later and link it to this issue.
Would love to see that feature, also to enable that globally for all tests, e.g. through cypress.json.
+1 for this feature, along with this it would also be nice to allow global and override retries in mocha it block.
@brian-mann Hi, do you have any updates on this?
@brian-mann Would also very much like an update about this.
+1 for this feature. We've seen some occasional failures in our CI that this would really help with.
This would be incredibly beneficial. I have a couple of tests that fail at random, and I have not yet figured out how I can get around this. But, if I can simply specify 5 retries, we can have slower yet useful CI builds ...
Quite obviously, even if we are writing Cypress tests idiomatically and correctly, there's no way to get around a flaky server or a flaky app. For this reason, we definitely need a way to retry individual tests. And I don't mind if it's a global toggle, e.g. to just specify max 5 retries for each test
This is by far the most needed feature for us right now. Screenshotting is low prio in comparison (I can hack that with blink-diff or use Percy if needed).
Without stable CI runs, e2e testing is not going to be too popular with my colleagues.
We discussed this today as a team. It may be the next feature @chrisbreiding works on after Firefox lands.
Great feature!
Can you already give some insight in when it will be available? :)
Also waiting for this. @brian-mann any updates?
+1
You can launch Cypress using the Module API , and effectively collect failed spec files from the run and tell cypress to run just those files. Here's an example:
https://gist.github.com/Bkucera/4ffd05f67034176a00518df251e19f58
Then, instead of cypress run, you use node cypress-retries.js
@bkucera That is a great and simple solution. Will give this a try. Thanks for sharing.
would love this feature to come out of the box for cypress: a global setting to retry failed tests x number of times to reduce timeout flakiness.
@brian-mann any news on where this is with the team?
@bkucera running v3.0.1, I had to change the
const specs = _(results.runs).filter("stats.failures").map("spec.relative").value()
to
const specs = _(results.runs).filter("stats.failures").map("spec.path").value(),
but it worked great for me.
I'm hoping for a way to rerun individual failed tests within a spec, but that didn't seem possible with the current setup, since Cypress parameters only included spec files. I'd love to hear any workarounds for that if you had any.
EDIT: v3.1.0 does use "spec.relative"
@bahmutov @brian-mann I think that with retries by workaround #1313 (comment) with Module API will not work with cypress 3.1.x with the parallel run.
The thing that scope of specs will not be same from original run to retry - because some tests will fail and some will pass. So run will fail with https://docs.cypress.io/guides/references/error-messages.html#Cannot-parallelize-tests-across-environments
@nanoflat I've updated the cypress-retries script to work with Parallelization.
The script should be able to work with parallel runs and non parallel runs. Just pass your configuration to DEFAULT_CONFIG
Here's how it will look on the Dashboard
@bkucera, your solution works great and I implemented it in my script.
Small problem, Cypress Dashboard marks the run as Failed while the reties succeded.
Is there a way to mark the run as Passed when retries have been successful?
Cypress didn't prove to remove all the 'flaky' test results for us, be we do enjoy the platform a bit more the Selenium webdriver.
The retry feature would be also a critical feature for our automation. We use similar mechanism for our selenium tests and it did save a lot of useless investigation time for false positives.
Hoping for a solution soon!
@Francis-Moreau-adsk Feel your pain, man!
Just my 2 cents:
(1) Are you sure the app being tested is robust? I've had flakiness issues too, and I could never pinpoint any exact issue. The back-end was a little slow when testing manually, but not inconsistent.
(2) Could it be that flakiness can be reduced by changing the way you write tests? E.g. avoiding await/async, or making tweaks according to Cypress Promise best practices.
Anyway, I am not using Cypress right now, but having excellent 'Retry' facilities will be important to me in the future.
@EirikBirkeland, thanks
The app we are trying to test is pretty solid but the way it's running there is one Cypress issue (#2552) that causes a lot headaches to write tests.
For sure we need to rethink the way we write tests, and we will get better as we continue trying Cypress. From my short experience with it, I don't see it as a good solution (for us anyways) for true end to end test scenarios. It appears to be viable solution to isolate a small ui component and test it very lightly. Even with the proper Promises in place, we will have to sometimes rely on other mechanism to synchronise the tests (reloads, gets, fixtures, waits...).
For more complex scenarios, there are a lot of shortcuts and workaround involved that real users will never have to do and I'm afraid some bugs might fall in the cracks if Cypress is the only solution to test the UI. Selenium webdriver will still show far more stable test results for us.
As for retries, they are dangerous to use and it's better to make the tests robust before even thinking of a retry mechanism. But I don't see how to completely escape them in a continuous testing build environment.
Still think Cypress is a beautiful beast.
@itaykotler-fundbox yes, the Dashboard will mark build as failed, and CI will say pass. Once we implement retries for real, that problem should be solved and Dashboard will pass along with giving you insights about your flakey tests 😄 and we currently have someone working on it.
Looking forward to having this feature!
This feature will be so useful! +1
I’m kind of confused with Milestones here on github. Seems this is in a closed sprint but not yet released? Is this currently being worked on or is it done? What’s the estimate?
seems like it got orphaned as they're on Sprint 13 now
Some initial work was completed on this, but there is still a good amount of work to be done. For the moment, the release of 4.0 and its features are being prioritized (it has some prerequisite work for this feature).
We recently started using milestones to track sprints (weekly work), which is why this is tagged with a closed milestone. I'm going to remove that milestone, since it's no longer relevant. When we start working on this again, we'll tag it with the latest sprint milestone. When this feature is complete, this issue will be closed and when it's released in a new version of Cypress, we'll post a message saying so.
@chrisbreiding thank you for the explanation 👍
I'm happy this will be worked on. Currently, I'm using if-then blocks to collect failed tests to be re-run at the end of a spec-file completed run.
@Phillipe-Bojorquez Do you have an example snippet that you wouldn't mind sharing?
@Phillipe-Bojorquez do you have the re run structure script in handy in gist or something .
One of the recent flaky failures we had was the login helper we use to auth pages through an API before visiting pages. Worked hundreds of times without failing, failing once caused the whole CI run to fail.
We'll look into the module API script for now. Still showing a test failing will help to know when/what is flaky. Sometimes it is a poorly written test, sometimes it is a complex system failing over something seemingly stable.
For anyone still waiting for the lovely Cypress team to get around to this, we came up with a solution we really like. We were using @bkucera's handy cypress-retries script, but it didn't work with our parallelized CI setup and we didn't like re-running the whole test file just to retry a single failed test, so we wrote this (somewhat hacky) little module:
if (Cypress.env('RETRIES')) {
let skip = false;
const _it = window.it;
const _beforeEach = window.beforeEach;
window.beforeEach = (...args) => {
const beforeEachFn = args.pop();
args.push(() => {
if (!skip) beforeEachFn();
});
_beforeEach(...args);
};
window.it = (testName, testFn) => {
if (!testFn) {
_it(testName);
return;
}
_it(`${testName} 🔄 1 of 3`, function() {
skip = true;
cy.on('fail', () => {
skip = false;
this.skip();
});
testFn();
});
_it(`${testName} 🔄 2 of 3`, function() {
if (skip) this.skip();
skip = true;
cy.on('fail', () => {
skip = false;
this.skip();
});
testFn();
});
_it(`${testName} 🔄 3 of 3`, function() {
const _skip = skip;
skip = false;
if (_skip) this.skip();
testFn();
});
};
window.it.skip = _it.skip;
window.it.only = _it.only;
}This simply monkeypatches it and beforeEach so that every test is tried three times. It obviously doesn't cover every use case but it works well for us. Just set the env variable CYPRESS_RETRIES=true & stick the code somewhere in your cypress support setup.
@debrisapron thanks for a hack, seems to work like a charm !
As for the feature this is necessary in our use cases !
@debrisapron Thank you! This appears to be working for me. 💟
I created a module that will (hopefully) be able to go away once retries are officially supported. I took the code @bkucera posted in a gist and made it a module.
Code: https://github.com/NicholasBoll/cypress-run
npm: https://www.npmjs.com/package/cypress-run
Install: npm install cypress-run -D
Change cypress run to cypress-run in node scripts.
Ha, @NicholasBoll I also wrote https://github.com/bahmutov/cypress-retry that retries failed spec but also modifies its source code to it.only the failed tests before rerunning. It is not polished, does not parse CLI arguments (all left as open issues for now), but could be useful as a starting point for PRs
@bahmutov That's pretty interesting using file rewriting to only run the spec that failed (as opposed to the whole file). You could probably just remove the non-failing tests rather than skipping them to decrease the noise.
cypress-run is meant to be a simple drop-in replacement for doing cypress run - simply replace the space with a dash for CI (the module publishes cypress-run as a binary entry point).
- cypress run --record --parallel --group ci --ci-build-id $BUILD_ID --key $CYPRESS_KEY
+ cypress-run --record --parallel --group ci --ci-build-id $BUILD_ID --key $CYPRESS_KEYDo you think it is worth combining efforts?
@NicholasBoll yeah, your script is really nice and just works. For my "retry" and for companion https://github.com/bahmutov/cypress-skip-and-only-ui I would justs keep them as novelties and things one can do completely in userspace. When actual built-in retries are added to the core of Cypress these modules would become unnecessary.
@NicholasBoll but we could definitely work on something useful, you know
@bahmutov Cypress does not run any it blocks if a beforeEach fails. Cypress assumes that if a beforeEach fails, then beforeEach will fail for all tests. Do you know if cypress-retry handles this case?
https://github.com/bahmutov/cypress-skip-and-only-ui is cool! I didn't think about using support to modify Cypress GUI elements. I might be able to implement Command Log Grouping in user-space to try out.
@debrisapron where did you put the script? I tried pasting into support/index.js, but I'm still getting my tests to fail after just 1 try (using both 3.1.3 and 3.1.4. I also replaced Cypress.env('RETRIES') with true to make sure it wasn't my env setup that was messing with things.
Also does it work in development mode with cypress open?
When you do it.only, this doesn't work, but that's OK!
Any success retrying a before() section?
Ha, @NicholasBoll I also wrote https://github.com/bahmutov/cypress-retry that retries failed spec but also modifies its source code to
it.onlythe failed tests before rerunning. It is not polished, does not parse CLI arguments (all left as open issues for now), but could be useful as a starting point for PRs
Hi @bahmutov how could run the code from your repo to check the retry?
Thanks!
For anyone still waiting for the lovely Cypress team to get around to this, we came up with a solution we really like. We were using @bkucera's handy cypress-retries script, but it didn't work with our parallelized CI setup and we didn't like re-running the whole test file just to retry a single failed test, so we wrote this (somewhat hacky) little module:
if (Cypress.env('RETRIES')) { let skip = false; const _it = window.it; const _beforeEach = window.beforeEach; window.beforeEach = (...args) => { const beforeEachFn = args.pop(); args.push(() => { if (!skip) beforeEachFn(); }); _beforeEach(...args); }; window.it = (testName, testFn) => { if (!testFn) { _it(testName); return; } _it(`${testName} 🔄 1 of 3`, function() { skip = true; cy.on('fail', () => { skip = false; this.skip(); }); testFn(); }); _it(`${testName} 🔄 2 of 3`, function() { if (skip) this.skip(); skip = true; cy.on('fail', () => { skip = false; this.skip(); }); testFn(); }); _it(`${testName} 🔄 3 of 3`, function() { const _skip = skip; skip = false; if (_skip) this.skip(); testFn(); }); }; window.it.skip = _it.skip; window.it.only = _it.only; }This simply monkeypatches
itandbeforeEachso that every test is tried three times. It obviously doesn't cover every use case but it works well for us. Just set the env variableCYPRESS_RETRIES=true& stick the code somewhere in your cypress support setup.
Hello, @debrisapron . I'm sorry about a dumb question, but to use your script, i just need put in .js and run switch like "npm run e2e-test CYPRESS_RETRIES=true" ?
I did that but there aren't any retry.
@vcamposs we are using the script and works great. We adding this to our Cypress.json:
{
...
"env": {
"RETRY_FAILED_TESTS": true
},
"video": false,
...
}
then your script that you copy pasted into your index.js can start like this:
if (Cypress.env("RETRY_FAILED_TESTS")) {
...
Also make sure you don't use any it.only commands
We're still working on this, but in the meantime I've made a plugin you can try out cypress-plugin-retries:
It supports mocha's this.retries, and it also retries failures in beforeEach
🚨 Do not comment on this issue about issues with the plugin 🚨
Let me know any bugs in the issues of the plugin repo.
@jennifer-shehane this would be such a great feature to help reduce flakiness in our tests. Is there an estimated timeline for this feature?
We're still working on this, but in the meantime I've made a hacky plugin you can try out
cypress-plugin-retries:
It's a similar to @debrisapron's solution but with a slightly cleaner UI, and it also retries failures in beforeEach hooks:
Let me know any bugs you notice in the issues of that repo.
Hello @bkucera ,
Seems the feature does not mark broken tests as failed and mark them as pending and hence you always have successful report after tests even some of them failed.
thanks @bkucera that plugin worked really well for us for beforeEach retries.
it also didnt create weird 'pending' result for the possible retries like another solution from above did.
so now i only have pass/fail. not pass/fail/skipped/pending which is much nicer.
@bkucera my retry setup just stoped to work, nothing has changed, just updated cypress to the last version, any ideas ?
@OgnjenPetrovic works fine for me
cypress@^3.3.1, cypress-plugin-retries@^1.2.0
Cypress.env(
`RETRIES`,
Cypress.env(`RETRIES`) !== undefined
? Cypress.env(`RETRIES`)
: (Cypress.browser.isHeaded ? 0 : 3)
);@bkucera, does it work with cypress cucumber as well? I added env: { "retries": 2} in cypress.json and import 'cypress-plugin-retries'; in /support/index.js, but I don't see any retries happening. Any thoughts please?
Is there any ETA on this feature, or estimate of roughly how much work remains?
This feature would make our continuous integration pipeline much more forgiving of random temporary network issues and other flakyness. 🙂
Bump on this. What is the status of this feature? Eagerly waiting for reducing flakiness!
^ I second that
Thirded 🙏
it's been roughly a year since the original feature request, any update on what's going on with this? Desperate for officially supported retries.
@germyjen don't think this is a top priority when there's a perfectly working workaround https://github.com/Bkucera/cypress-plugin-retries
Hello there,
I wanted to chime in and try to give you an understanding that this is currently a Number 1 issue and is actually costing you money:
We just had to switch from our 5000$ a year cypress plan to our own parallel implementation. This is due to the issues described above: You can't retry a cypress test reliably in a pipeline when using the parallel feature.
For us this even meant that once you retried a cypress test it always went "green", so we actually deployed failing stuff.
So this issue alone will cose you 5000$ a year now. I would be "SO" happy to pay you for your services, but as it is now, we cannot use your commercial product, even though we want to.
not trying to minimize the importance of getting this in Cypress core. We hear you, it's up next to be worked on.
But since Github has started collapsing comments in this thread I wanted to make sure people see the workaround plugin posted above:
We're still working on this, but in the meantime I've made a plugin you can try out
cypress-plugin-retries:
It supports mocha's retries, and it also retries failures in beforeEach
🚨 Do not comment on this issue about issues with the plugin 🚨
Let me know any bugs in the issues of the plugin repo.
@bkucera this is not really a workaround, because one can still manually retry a test in e.g. GitLab und force it to pass, even if there are real errors that where introduced with this test.
Furthermore it only retries them automatically. Chances are high that the test failure was introduced by a third party system not being available. The plugin would not do anything for that kind of failure.
The plugin works well for us. @Algram if your tests are dependent on third party systems, you might re-think them... it's generally best practice to mock out external dependencies that you have no control over.
I'm facing the same issue with @Algram. I'm trying to run cypress parallel on Bitbucket Pipelines and it's always green when you rerun failed step. I tried cypress-plugin-retries but it didn't help.
any news on this?
It looks like the Cypress team is building out native support for retries, so stay tuned!
Would recommend giving the feature an up vote on the Cypress roadmap: https://portal.productboard.com/cypress-io/1-cypress-dashboard/c/59-automatically-retry-failed-specs
The code for this is done in cypress-io/cypress#3968, but has yet to be released.
We'll update this issue and reference the changelog when it's released.
Test retries has been released in 5.0.0.
You can refer to our docs on Test Retries for instructions on how to turn on and configure test retries.
There are several options which we think are important to explore depending on your project including configuring globally, per cypress open and cypress run separately, or per test suite / single test.
We think this will be a valuable tool in helping identify flake and have more plans in the future to improve this feature and surface the test retries as well as analytics on flake into our Dashboard.
If you encounter any issues using Test Retries, please open a new issue with a fully reproducible example - do not comment in this thread as this issue is closed.
Awesome work @jennifer-shehane and team, you're the best <3
for anyone coming from the retries plugin:
Migrating from cypress-plugin-retries to Cypress 5.0.0:
- remove cypress-plugin-retries from
devDependenciesand related code in support/plugin files - To enable retries on single test/suite, remove usage of
Cypress.currentTestin favor of test config overrides e.g.:
// on a single test
it('test title', { retries: 2 }, () => {
...
})
// or on a suite
describe('suite title', { retries: 2 }, () => {
...
}) - To enable retries globally, set
retriesincypress.jsoninstead of usingCypress.env('RETRIES')e.g.:
{
"retries": { "openMode": 0, "runMode": 2 }
}- remove usage of
this.retries(n)(not supported)
The Dashboard now displays the screenshots and errors from each attempt of a test that retried. The test results are also filterable by whether they retried.
Filter by whether test retried
See screenshot and error for each attempt when clicking on test
retries: 2
Given('the user is on accounts location page', () => {
{ retries: 3 }
cy.get('#search').click().type('test').type('{enter}').wait(1000); --> if this fails I need to be able to retry this 3 times same step till the actual thing loads
cy.get('table').contains('a', 'test').click().wait(3000);
});
@jennifer-shehane @bkucera
How can I use that object in cucumber plugin ?
Also my current issue is that when I use cy.wait(3000) if the element or request is not back by then it will fail the test. what I wanna achieve is to retry three times same steps if it failed at 3 seconds time out. is there anyway of doing this?