simonsmith/cypress-image-snapshot

Diff output for each attempt

axelboc opened this issue · 5 comments

axelboc commented

Hi, I'm looking for a way to generate a diff output image for each failed attempt and not only for the first one.

Sometimes snapshots don't match because the testing environment is just "warming up". I've configured Cypress to retry the tests a few times to alleviate this issue. If a test fails on the second attempt, then it's more likely to be because of a regression or expected change. So the diff output I want to look at is rarely the one generated on the first attempt but more likely one generated on a subsequent attempt.

simonsmith commented

Hey @axelboc,

This sounds like an issue that should be solved at the application level, not this plugin. Particularly because you use the phrase "warming up". Your application should be ready and in a stable state before any screenshots are taken. Note this section from the documentation:

Taking a screenshot is an asynchronous action that takes around 100ms to complete. By the time the screenshot is taken, it is possible something in your application may have changed. It is important to realize that the screenshot may not reflect what your application looked like 100% when the command was issued.

https://docs.cypress.io/api/commands/screenshot#Asynchronous

For example I often have to assert some elements are visible before I begin the matchImageSnapshot() call, something like:

cy.get('.some-element').should('be.visible')
// now perform snapshots

I've got this plugin running across six large applications, most SPA and I have to ensure they are all stable before screenshots start and not in a loading state/rendering phase etc

axelboc commented

I wish I could, but I'm working with a WebGL canvas (via react-three-fiber) and I have yet to find a way to assert that the canvas is fully painted before taking the screenshot... 😭

I have found a compromise between delaying the screenshots and not slowing down the tests too much, and it works well most of the times. When it doesn't, Cypress' "retry" feature saves the day. Unfortunately, when the canvas is too slow to paint while there's also a regression or expected change to the screenshots, then the diff image often hides the regression/expected change.

I have increased the wait timers for now (and I can still sleep at night 😂) but it would still be cool to have a way to inspect the diffs of all the attempts.

simonsmith commented

Ah I see, yes canvas is a tricky one. Not sure if this article could be of help, depends how complex your scene is.

To revisit your original question, if a diff image was generated for each retry would that just be for debugging? I can't see how it would make the test less flakey unless I've misunderstood

axelboc commented

Oh, it won't make my tests less flaky for sure, sorry this wasn't clear. It's so I can properly identify whether the screenshots have changed due to a meaningful change or a regression, and therefore decide whether to approve the change and run Cypress with --env updateSnapshots=true.

Basically, I have a CI job that fails when there's a diff to the screenshots. When this happens, I download the job artifacts and look at the diff. If I agree with the change, I comment with "/approve" on the PR, which triggers an "approval" workflow that runs Cypress with --env updateSnapshots=true and opens a PR with the updated screenshots.

Right now, when there's a regression/meaningful change, the diffs of my flaky tests sometimes show a blank canvas. In order to be able to review the real change, I have to either:

  • run the Cypress CI job again until the flaky tests work and show the meaningful diff (which requires downloading the job artifacts each time to check the diffs)
  • "approve" the snapshots to run the approval workflow and re-run that workflow until the screenshots in the PR no longer show a blank canvas.

The second solution is the least tedious, but the risk is to inadvertently "approve" a regression, which requires closing the PR opened by the CI, fixing the regression and "approving" again. This can be quite noisy for people watching the repo.

If I could look at the diffs of all the attempts right away, I could ensure to only ever approve meaningful changes.

Thanks for the link! I do like the idea of a waitForStaticCanvas, a la waitForStableDOM. I'll definitely look into it. 😊

  • 👍1
simonsmith commented

So the ideal solution to this problem for you would be the plugin saves each snapshot it took somewhere for each retry so you can look back at them? Right now that is purposely overwritten so that each retry does not add a new file to the filesystem (and allow the tests to just pass).

I'm imagining a scenario where there could be some sort of option to enable this behaviour. However I'd probably pursue the approach in that article before relying on this 😄

  • 👍1