Capture + Display all logs for everything that happens in Cypress
Closed this issue · 92 comments
Problem
Many of our users express the desire to see console.log statements when running headlessly. Typically you'd be able to see these when you're working in Dev Tools (in headed GUI mode).
We can pretty easily display console.log statements by mixing them into stdout when running headlessly, but we don't think this is really a "true" solution. It's kind of a hack and it still doesn't really tell you "everything" that's happening. For instance, when a console.log happens, how would you know when it happened in relation to everything else that's going on in a test?
So, we think to truly solve this problem, we need a comprehensive way of displaying not just console.log statements but also capturing:
- Network Traffic
- Commands
- Retries
- Under the hood Cypress wizardry
- DOM snapshots on failure
By providing all of this, the entire picture of what happened during a test is revealed and now you actually have even more information at your disposal than even what the GUI headed version provides you.
Solution
The logs will be available in the Cypress Cloud.
Once we capture and parse these logs, it will unlock an enormous amount of power in Cypress. We'll also be able to analyze the logs and deliver things like:
- Analytics into your tests
- Comparing a failing test to a previously passing one
- Diffing the logs for changes in response bodies, etc
- Understanding why tests are slow
- Discovering command anti patterns (like explicit waits in your code)
The logs could also be interactive - for instance they should "sync" up to the video and enable you to playback everything that actually happened in a Cypress test.
Here is an example comp of what we're going for.
We're still working on communicating the start and end of logs, and also communicating that a log event acts as "a stream" but that events are connected to other events.
For instance when a HTTP request starts - that's a 'start' event. When the response comes back, thats a separate 'end' event. However these are connected and you likely need to be able to view the data for both events from either the start or end event.
Another example:
A command starts. It retries 65x times. It eventually finishes. The duration of the retry events, the reason its retrying, and the eventual delta between a start and an end needs to be communicated intelligently.
These log files end up being a massive amount of data, but we already have the infrastructure in place to capture, parse and deliver them. At this point we're just iterating on a UI that's intuitive and doesn't overwhelm the user.
We're going to be providing a new Logs tab, but also for individual test failures, we'll just be providing the logs for that one independent test.
Feedback
...is welcome ;-)
@bahmutov what do you think?
Perfecto! 😍
@brian-mann excellent. Really looking forward to getting this. We're spending lots of time trying to debug why our compiled JS is not working when served static via Cypress. We cannot replicate anywhere else, so one peek at the logs would like solve everything!
Hi guys!
Amazing feature! Any news about that?
Thanks
AMAZING!
@brian-mann I've been banging my head against a wall trying to upgrade to cypress 0.20.1. It's been days - things work fine in the GUI but fail from the console. And not being able to log anything is infuriating. After super old school commenting out code one line at a time until it passes again to isolate the line, I've found that it is cy.request that fails. But I don't know how to fix it yet, and I can't log anything so have no idea what the issue is. Adding all the features you noted in this ticket sounds fine, but we need a simple way to debug locally, in the command line itself. How can we get console.log (or cy.log) to show in stdout when running headlessly? We're blocked from using Cypress right now until we fix that because our CI is failing now due to this other error, which is supposedly fixed in 0.20 #517
@lukemadera have you tried using DEBUG variable to see messages from Cypress? https://docs.cypress.io/guides/guides/debugging.html#Debug-the-Command-Line
- Is it failing locally or only in CI?
- If failing locally be sure to run in
--headedmode so you can see what's going on. - You can also switch to using
--browser chromefrom the CLI (in local and in CI) if it fails only in electron. Electron is technically a different browser from your normal Chrome. - The key here is getting it to be reproduced locally. You can try running Chrome from the GUI, but using the Dev Tools to throttle back the network, and also decrease the CPU so that it mimics a much slower computer. That can sometimes highlight a race condition you are not accounting for.
- As per getting events out of the browser - that is not necessarily a straightforward thing. What exactly would you like us to send to
stdout? Internal information about what Cypress is doing? You will see hundreds if not thousands of events being fired over the course of a run. And that's just what Cypress is doing. Nothing from your application like network requests, console output, etc would be sent unless we chose to do that. - I would like to run an experiment here. Open up Cypress in GUI mode and then open the console and follow the instructions here: https://docs.cypress.io/api/events/catalog-of-events.html#Logging-All-Events.
That will send every internal event that Cypress fires to the console. Look at this and tell me if it is helpful. If the answer is yes, we could pretty easily get that out of the browser and into stdout. If it is not helpful then we'd need to figure out what is. Browsers are enormously complex and trying to distill what's happening down into text is not that simple. What exactly do you want to see? What would be helpful?
Also one last important piece of information. Why aren't you just recording your runs? Even though its headless you can see exactly what is going on since we capture a movie and screenshots. What about that information is not enough to see what's going on and be able to reproduce locally?
Or at the very least why aren't you watching the video from headless runs or looking at screenshots?
@bahmutov Yes I've tried debug. That gives general info, not info I generate and want to see.
@brian-mann thanks for the speedy response. There are 2 failures / issues right now.
A. on CI, v0.19.x no longer runs (headless)
B. locally, v0.20.x runs in GUI but fails in headless.
-
All the tests pass in the GUI so unless I'm missing something, the error won't happen in headed mode, so I can only debug in headless mode.
-
Again, it's a general request issue (I think - still need logging), not a browser issue.
-
I can reproduce it locally 100% of the time, in headless mode. Works 100% of the time in GUI mode. These are tests that worked fine for months, and they work in v0.19. Just not when upgrading to v0.20.x
-
I'm simply looking to be able to do
console.log()and have it show up on the command line when I do./node_modules/.bin/cypress run --spec cypress/integration/locations/locations-list.js -
while all that data MAY be helpful, right now I just need ONE (or a few) pieces of data, that I choose from
console.log() -
I'm not even getting to the screenshots or videos really, and I've already figured out the problem (it's in our data setup phase, trying to login via cy.request). It's not an actual test that is failing, we're not even making it to the tests. So I'm not sure how videos / screenshots would help. It's literally the first step that's failing, in setup, before tests even start.
If this is still unclear I can record a video of the test in the GUI and the test in the command line - one finishes and passes, one just gets stuck and never does anything (or times out).
Update: here's the video: http://recordit.co/DdfW8hgK74
Okay let me clear up a few things.
When we say GUI mode we're talking about launching cypress from cypress open. In that mode the tests never "complete" and you don't get anything on stdout.
When you run from the CLI via cypress run then the tests complete, you get stdout and an exit code.
Cypress itself does work differently in these two modes. When running from the CLI it makes internal optimizations to be able to run all the tests - ie it will run faster than the GUI mode. The GUI mode is focused on debuggability and interaction.
My point is this: using --headed from the CLI simply shows you the browser. It will behave identically to running headless. One shows the Electron browser window, and the other doesn't. That's it.
In fact you could just launch Chrome via the CLI with --browser chrome and then you'll be using the same two browsers in GUI mode and in CLI mode with the only difference that one runs to completion, the other doesn't.
The screenshots and video would still help in this situation, albeit I understand the need for console.log. At the very least they would show you precisely which step and command is failing without you having to guess in the dark.
Update: per your video - just add the --headed option or use --browser chrome to see what's going on. If you choose not to do that, you could still use the video Cypress captures when you run headlessly so you could still see what happened in that mode.
Also at this very moment you could do this trick here: #300 (comment)
That will get all console.log's to show up in the command log, which will stringify and print their arguments. You could then watch the video / screenshot to see what the information is.
That's what I would do today when running in CI. Of course locally just show the browser and iterate on it there.
I opened a couple new issues to do what you're asking with exporting the console output, and some improvements to the debugging the driver.
YES! The --headed works and I can see console.log output, thanks so much @brian-mann ! It does still close automatically at the end, which means I can't see the logs persisted, so I have to add a wait at the end. Any way to prevent the browser from auto-closing?
However, I'm still stuck at the same cy.request line. It just doesn't ever complete. In digging into it, cy.request works fine if it's called BEFORE cy.readFile. If it's used after (within the .then()), it just never returns.
In trying to reproduce it, if I simplify things it seems to work, so I'm not clear on what the issue is.
If I just do a request (using the npm request module instead of cy.request, I get a CORS error).`
Luckily, the request I'm attempting is to logout so in this case I can just skip that request all together, and then things work fine. Now tests appear to be passing again.. Locally at least. Going to try in CI now.
Not sure if the issue is "fixed" but it's working for me now it seems. Thanks for all the speedy responses and help!
If you want to iterate without it close, just open up Electron in the GUI. Just select that browser instead of Chrome. You may see the same issue.
As I mentioned if that's the case just switch to booting chrome from the CLI --browser chrome
This would be a really nice addition. Right now my tests are failing on the CI and it looks like it has something to do with cookies but because Cypress is basically a black box right now on CI I have no way of figuring out what's going on. Hope this feature will arrive soon, keep up the good work 👏🏼
Just want to throw my 2 cents in, I'm also having an issue with (what I can only imagine is cookie-related) the CI build vs local cypress open run that @rovansteen is having, and not having a way to see what network requests are made is making it challenging to debug.
So far I'm loving Cypress though, you folks are doing amazing work and I'm more than willing to fork over our company credit card for a paid plan in the future!
Hi, loving Cypress. But I am curious. Are these issues still being addressed?
I see a lot of comments that these issues are being worked on, like: "x and y. Those will land pretty soon". But this seems to have been in development for nearly a year now.
Respectfully, if we can't debug our CI than this isn't really a viable product for the long term. And we would rather know sooner than later since E2E tests are very costly.
@brian-mann do you have any update on this one? In October you said soon. I know your backlog is very long but CI logs are quite important. Every time there is an issue on the CI we need to run the tests again locally in order to get the logs.
Thank you for your understanding 🙏
I'm wondering if anything is happening here, it's way too quite for a feature all projects need at the point where they have to maintain tests and the CI console output and the test results can't tell what went wrong except the last exception message.
Sounds like a hack but if you
- start an HTTP server in the background with a log endpoint (the endpoints simply writes everything on its console)
- patch the cy.log() command to make a request to your endpoint with the arguments it got
You should at least see the cy.log messages. It would be much more elegant if it goes through the existing cypress internal websocket though.
Anyone here tried something like this before?
For everyone who wants to get the cy.log and other Cypress commands this might be helpful https://github.com/bahmutov/cypress-failed-log
First, thanks for all the good work!
But I would address this particularly issue again.
To give a daily example. We couldnt reproduce a 'bug' in our application in local development and our CI server continuously failed. We were not able to find why, because we were not able to inspect the console errors (with an error when a network request failed). After a lot of research we discovered that our CI environment was not able to connect to a specific endpoint.
So you cannot guarantee that cypress test running locally will behave exact the same as on your CI.
It was a little bit frustrating, because in the summary of this ticket there was a line
We can pretty easily display console.log statements by mixing them into stdout when running headlessly
and that simple functionality would helped us so much.
As kind of a hack we now have a feature flag when running E2E on the CI that when having a console error, the page will output the error.
/* This part is to put console errors in the HTML when running cypress tests */
var console = (function(oldCons){
return {
log: function(text){
oldCons.log(text);
},
info: function (text) {
oldCons.info(text);
},
warn: function (text) {
// $FlowFixMe
document.getElementById('root').innerHTML = text
oldCons.warn(text);
},
error: function (text) {
// $FlowFixMe
document.getElementById('root').innerHTML = text
oldCons.error(text);
}
};
}(window.console));
if (window.envConfig.cypressConsole == 'true') {
window.console = console;
}Since 3.0, we're collecting more data related to the timings, body, and hooks run for recorded tests.
This issue has been put on hold as there was a great deal of work to be done in the server to gather all of the data necessary. It is still on our roadmap for the future - and there has been some work done on it.
+1 for this being implemented
Yeah, if it's so easy to mix cy.log into the console output, then give us a CLI config option for outputting that to the console. Don't just leave us helpless for more than a year working on the perfect solution.
Hi,
Is the dashboard logging tab still in the development plan?
We just had this issue, where tests pass locally, but fail on CI. We have no way of knowing why they fail as we just get a White Screen of Death and no console.log output. I'm at a loss for any way to know what error is being thrown as I can't even reproduce it locally. Any other tips for jury-rigging console logging on Travis?
This feature is part of our Dashboard Service Roadmap, and it is now under active development.
@amirrustam Does it mean it will be available only when using Dashboard (and not when using standalone cypress run without test recording)?
@kamituel we've considered both use cases. No additional details available at this time. So stay tuned. Generally speaking, using the Dashboard enables ancillary benefits that are geared towards pin-pointing test failure root causes and faster debugging.
The dashboard is not available during automated testing runs, such as running Cypress on Travis-CI, where it is critically important that there is a way to be able to log output to stdout so that it's possible to track where in a script things start to go wrong.
Can we still get something like cy.stdout.write?
Hey @Pomax
The dashboard is not available during automated testing runs, such as running Cypress on Travis-CI
Could you explain a bit more by what you mean in the statement above? Is your company not able to record to the Dashboard? I know there have been requests for On Prem due to security restrictions, for example.
Having the ability to access the raw logs is certainly something our team is considering.
We're using Cypress during our Travis-CI passes, so Cypress gets run in a fully contained test environment with only "stdout output" as viewable data. Cypress gets to run the interaction tests, and take snapshots, but without access to any kind of external app. because it needs to be able to do its work irrespective of whether "the rest of the internet" is up or not.
I don't really want "access to logs", if those are log files written to some particular location, because the nature of CI is that you get to run code, and only run code. Everything is thrown away the moment testing is done, and the only thing that gets preserved is the console output.
So there really should be a way to say something like , for instance, cy.console.log("...") or cy.stdout.write("...") or something, so that people can get data written to the running console output directly.
Compare this output to the next:
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 3.0.3 │
│ Browser: Electron 59 (headless) │
│ Specs: 2 found (multi-page.js, single-page.js) │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: multi-page.js... (1 of 2)
multipage visual regression tests
2019-02-01 09:51:41,051 [INFO] "GET /en/campaigns/multi-page/ HTTP/1.1" 200 13332
2019-02-01 09:51:41,051 [INFO] "GET /en/campaigns/multi-page/ HTTP/1.1" 200 13332
2019-02-01 09:51:41,182 [INFO] "GET /_images/green-20.svg HTTP/1.1" 200 3459
[...snip...]
2019-02-01 09:51:41,510 [INFO] "GET /environment.json HTTP/1.1" 200 235
2019-02-01 09:51:41,510 [INFO] "GET /environment.json HTTP/1.1" 200 235
√ Loads the multipage campaign correctly (4415ms)
1 passing (4s)
(Results)
┌─────────────────────────────┐
│ Tests: 1 │
│ Passing: 1 │
│ Failing: 0 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 0 │
│ Video: false │
│ Duration: 4 seconds │
│ Spec Ran: multi-page.js │
└─────────────────────────────┘
The above run tells use nothing about where in the test things are happening. Being able to effect something like the following (without resorting to custom plugin interactions) would be far more valuable to have when doing automated testing (which is of course the whole reason people start using Cypress: they have testing to automate).
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 3.0.3 │
│ Browser: Electron 59 (headless) │
│ Specs: 2 found (multi-page.js, single-page.js) │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: multi-page.js... (1 of 2)
multipage visual regression tests
2019-02-01 09:51:41,051 [INFO] "GET /en/campaigns/multi-page/ HTTP/1.1" 200 13332
2019-02-01 09:51:41,051 [INFO] "GET /en/campaigns/multi-page/ HTTP/1.1" 200 13332
[CYPRESS:multi-page.js:21] Page loaded, waiting for React bundle...
[CYPRESS:multi-page.js:24] React bundle loaded
[CYPRESS:multi-page.js:28] Running Signup Form interaction tests
2019-02-01 09:51:41,182 [INFO] "GET /_images/green-20.svg HTTP/1.1" 200 3459
[...snip...]
[CYPRESS:multi-page.js:112] Testing 2FA authentication dialog
[CYPRESS:multi-page.js:118] ERROR: undefined is not true
[...snip...]
2019-02-01 09:51:41,510 [INFO] "GET /environment.json HTTP/1.1" 200 235
2019-02-01 09:51:41,510 [INFO] "GET /environment.json HTTP/1.1" 200 235
√ Loads the multipage campaign correctly (4415ms)
6 passing (4s), 1 failed (4s)
(Results)
┌─────────────────────────────┐
│ Tests: 7 │
│ Passing: 6 │
│ Failing: 1 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 0 │
│ Video: false │
│ Duration: 4 seconds │
│ Spec Ran: multi-page.js │
└─────────────────────────────┘
Using the dashboard for "in house" or local dev work is perfectly fine, but paid automated CI testing means everyone is running cypress without a dashboard. Not out of spite, and not because "they hate having to use a GUI" but because they literally can't run any graphical applications because the testing environment itself is fully automated, and everything in it gets thrown away (which a good CI service should be doing) leaving you just with the CI run console output to see what went wrong, if something went wrong.
OK, while we wait for this perfect solution described in OP to arrive, I would really like to vote for a very simple solution of redirecting all browser console to the terminal during CI run. It is really hard to debug anything currently when it fails in CI. I would love to simple see in terminal all logging calls I might have inside tests themselves and also from my app. I mean, we already have those logging calls for debugging with a browser, let's just log them to the terminal as well.
Let's not make the perfect the enemy of the good.
@bahmutov created a plugin that will print Cypress commands to the terminal - not exactly the browser console logs wanted, but maybe will help someone. https://github.com/bahmutov/cypress-failed-log
We've also been expanding on our Debugging doc - adding some weird tricks in there that could help some people. https://on.cypress.io/debugging
Adding to the general commentary above: we've also run into this, namely the lack of full Cy-related transactions not being made available to some target (like stdout) when running headlessly/headlessly in CI (Jenkins).
In our case. where this really becomes a pain is when things pass "locally" - but fail in CI, or fail in Docker Compose scene (and running in a container with open instead of run is already classed as "not supported").
The sooner this gets introduced - even in some MVP form - the better.
First apologies for the comment which is not trying to fix the actual problem. 🙏
Then let's say cypress is really cool and I really liked to use it 🙌
However, I can't suggest / use it to anyone until this is fix it.
The workaround is fine but far from being a good solution.
Being by default blind in CI is a big deal.
What I now do is keep cy.logs really short so that I can hopefully read them from the screenshots that are available on the Dashboard. But it is a silly solution of course. It would be so helpful if the cy.logs would be available on the Dashboard
I want to assert that the Console log is empty, or only contains previously approved lines in the CI build. The optimal implementation would give a simple textfile with timestamped direct copies from the console output.
These logs could be prepended by the testcase name and current URL for context, adding more stuff would be noice.
As others have commented, it feels rather bad that the console log and Cypress command log can not be saved to files (separately or together, by configuration) 2.5 years after the issue (#300) came up.
It's been 2.5 years since this was brought up and the fix everyone wants was declared "easy." All of the suggested fixes around using the dashboard are problematic for reasons described thoroughly in this issue. Is there any movement on this?
Current workarounds: use these 2 plugins
Just stumbled upon this need ):
https://github.com/flotwig/cypress-log-to-output workaround only works on Chrome. But chrome headless does not work and does not support video recording... its going round and round the loop for a very simple request (i.e. proper console,log)
I don't see the feature listed in the Cypress Roadmap anymore and the two suggested workarounds aren't ideal, at least for me. Is there still planned work for this? We're fully integrated with Cypress and Cypress Dashboard but we're having trouble debugging issues without having the console.log available when running tests via CI.
any update on how we can get console.log showing when we use "cypress run" ?
e are thinking of moving to cypress and this can be a show stopper :(
In case you're using Electron browser you can see your console.log() statements with DEBUG=cypress:electron npx cypress run. More on it here. For Chrome the abovementioned cypress-log-to-output plugin should help
Any updates on this? I have a cicd test that fails, that I can't reproduce locally, and I'm burning through my Github Action minutes.
This issue is still in the 'proposal' stage, which means no work has been done on this issue as of today, so we do not have an estimate on when this will be delivered.
Current workarounds: use these plugins
- https://github.com/flotwig/cypress-log-to-output - Prints all browser console logs to the terminal while running Cypress tests. Currently, only Chrome is supported.
- https://github.com/bahmutov/cypress-failed-log - Saves the Cypress test command log as a JSON file if a test fails.
- https://github.com/archfz/cypress-terminal-report - Logs cypress commands, route request data and browser console errors and warnings to terminal when tests fail on CI.
I'd like to write my own little helper for this, what command do I use to print arbitrary strings to the CLI when using cypress run? Or is that the underlying issue here - that there is none?
Jumping in on the bandwagon calling for better logging especially on failure when matching json objects. I'm looking for a test framework for REST API testing where the lack of logging/reporting prevents me from choosing cypress
Jumping in on the bandwagon calling for better logging especially on failure when matching json objects. I'm looking for a test framework for REST API testing where the lack of logging/reporting prevents me from choosing cypress
That's not really the usecase Cypress seeks to meet. If you want to test APIs, you should probably be looking at Postman.
Thank you for the link. I am a long time postman user, however I didn't know about their automated test capability. Thank you
@brian-mann any updates on this?
Hi @brian-mann could you make any comment on this ticket ? Would be great to have some news on its progress since it was raised a long time ago (March 2017). Thank you very much.
I really like Cypress because of the attention put on developer experience so I will share my feeling hopping it will motivate / bounce this issue:
As a DevOps developer I can tell that this feature will considerably enhance my experience, for the moment I need to use "cy.writeFile" to collect all logs and then I need to download artifacts to see this written file, which is relatively painful...
Would really love an update on this feature request as well. @amirrustam
This is so frustrating .... anything new on this one?
Do we have news on this? It would be awesome to add it.
When is this feature coming. We really need it.
Thank you.
➕ 1
+1
+9000
I need this feature. Been waiting on it.
+1
➕ 1
I am amazed this is still not a feature of Cypress tbh. Is there really no way to show cy.log() entries in the official results dashboard?
@Phonesis This may not be the exact thing you are looking for, but the way I display cy.log() entries in dashboard is by appending them to error message, you can find the code snippet at the end of this blogpost
@filiphric Thanks for the info this pointed me in the direction where I could at least log out what i needed from our CI build agents. I needed to get the timezone info so simply logged it to an error i threw in my code directly:
throw new Error(
`time was ${timezone?.id ?? 'unknown'} ${timezone?.name ?? 'unknown'}`
)
This obviously shows in the dashboard as it's an error.
It really would be nice to be able to log stuff though and see it referenced in the dashboard. Or even better... a way to show the browser console.log in the dashboard.
It's now been over 5 years and this is still not a feature. Being able to view the console.logs (and such) would extremely help in debugging failing tests.
Yep. I need this right now, it's crazy this isn't a thing in Cypress.
There are tons of workarounds in this thread alone, have you considered any of them?
That would be awesome, also if we could group commands such as cy. session, that would be beneficial as well. :)
➕
A lot of work happened over the last little while to enable us to finally work on something like this. It's not a lot more practical to implement something like this due to various under-the-hood improvements to the App and Cloud platforms over the last year or two. I hope we can share some updates and progress soon.
In the meantime, some debugging-in-CI techniques I've found useful:
Is there really no way to show cy.log() entries in the official results dashboard?
The way I've done this historically is by using cy.log and cy.screenshot. You can do cy.screenshot(..., { capture: 'runner' }) to include the command log in the screenshot.
You can pass JSON.stringify(<value>, null, 4) and get some pretty elaborate logging going. Not really a fully featured solution, but it's helped me out a ton when debugging.
Screenshot docs: https://docs.cypress.io/api/commands/screenshot#Arguments
I am looking for this as well, and none of the workarounds are usable for me.
My app outputs various information to the console while in debugging mode, which is enabled for Cypress.
In a test, I would like to do something simple such as click on a button, then make an assertion based on what was immediately logged to the developer console afterwards.
The only actual workaround for this is to stick a hidden div somewhere in the DOM that mirrors everything that is being output to the console. My apps communicate internal states and variables for tests, and these are not always accessible from the DOM itself. I would like to assert based on these console.log messages.
This issue is still in the 'proposal' stage, which means no work has been done on this issue as of today, so we do not have an estimate on when this will be delivered.
Current workarounds: use these plugins
- https://github.com/flotwig/cypress-log-to-output - Prints all browser console logs to the terminal while running Cypress tests. Currently, only Chrome is supported.
- https://github.com/bahmutov/cypress-failed-log - Saves the Cypress test command log as a JSON file if a test fails.
- https://github.com/archfz/cypress-terminal-report - Logs cypress commands, route request data and browser console errors and warnings to terminal when tests fail on CI.
@brian-mann @bahmutov @lmiller1990 Any official work being done on this feature? I will try the plugins mentioned here. I would also love to keep in step with future improvement. I think this is an immensely valuable feature.
@M4RcB4 Yes, there is. I don't have an exact date, but it's an active and high priority project. Please stay tuned - more info will come as soon as it is ready for testing!
The goal is to capture all the things - console, network etc. As a long time user, I am quite excited - debugging with videos is good, but this will be better.
Hey, is the intention for the logs to be only available via cypress dashboard, or would they also be printed to the terminal/saved to a local .txt as well?
The project that @lmiller1990 mentioned will be only available in the cypress dashboard.
So, still no way to make assertions against console output.
@Opalexiss you can definitely make assertions against the console methods: https://glebbahmutov.com/cypress-examples/recipes/check-console-logs.html
Closed by #27040
@chrisbreiding nice, thanks! Although, the PR description on that #27040 PR is a bit unclear - which PR in v13.0.0 actually addresses this issue #448?
I couldn't figure that out from either the PR description or the v13.0.0 changelog
@karlhorky I can understand how it might be a bit confusing. #27040 was a PR for a long-running branch that collected a number of PRs for the v13 release. That includes several PRs that address this issue. For the most part, however, those changes are not user-facing, so they're not called out in the v13 PR or the changelog. Most of the work for this is under-the-hood (capturing all the things!).
While the changelog published with this repo doesn't mention it, the changelog that will be published to the docs highlights the functionality this will enable for Cypress Cloud.
Released in Cypress 13.0.0 as Test Replay. Please read our Blog for more details.
Nice, thanks for the additional background, looks exciting!
@jennifer-shehane I am not sure Test Replay was the actual requested feature.
Personally, I want to be able to dump console.log, network tabs to a file, so we I can read it to understand what went wrong.
Best,
T.C
@jennifer-shehane what's the story for replay around non-cloud CI/CD setups?
I'm a bit worried this is a dark pattern change designed to remove useful functionality from that set of users, especially when viewed as coupled with the removal of videoUploadOnPasses, for instance. Happy to take the tin foil hat off, but I didn't see any discussion here or on other threads of allowing an "asset" to be captured in CI that would let people view the runs locally.
@comp615 Was something removed? I don't work at Cypress, but I still use it a lot -- my understanding is the Test Replay is for Cloud users. You can still record videos to your local machine, review them after a failure, as part of the OSS offering.