simonsmith/cypress-image-snapshot

cy.matchImageSnapshot() failing in CI

awnclaim opened this issue · 6 comments

awnclaim commented

Hi there,

All my specs fail when run in GitLab CI with the following error message:

then function

Here is a spec example:

describe("Visual regression tests for Idea Tasks", () => {
  const sizes = [
    [1920, 1080],
    [768, 1024],
    [390, 844],
  ];

  const scrubbedElements = [
    ".one",
    ".two",
    ".three"
  ];

  sizes.forEach((size) => {
    it(`Idea Tasks when '${size}' resolution`, () => {
      // custom command
      cy.setResolution(size);

      cy.visit("");

      cy.wait(1000);

      cy.matchImageSnapshot({ blackout: scrubbedElements });
    });
  });
});

It's okay locally, and the specs not using the cypress-image-snapshot plugin work fine in CI as well.

Thank you in advance!

simonsmith commented

That would likely indicate that the snapshot hasn't matched, if it's failing in the then callback. I'd recommend this plugin to give you better feedback in CI

What does the console output say currently?

awnclaim commented

Hi Simon, thanks for your swift reply. With the plugin you recommended, the output isn't really clearer:

      cy:command ✔  wrap	
      cy:command ✔  task	Matching image snapshot, Object{13}
      cy:command ✔  screenshot	Visual regression tests for Idea Tasks -- Idea Tasks when '1920,1080' resolution
        cy:fetch ➟  GET ...
                    Status: 200
        cy:fetch ➟  GET ...
                    Status: 200
        cy:fetch ➟  GET ...
                    Status: 200
      cy:command ✔  task	Recording snapshot result
      cy:command ✘  then	function(){}

Also, normally, when the snapshot hasn't matched, I get screenshots with the word "diff" in the title, and in this case there're no diffs.
image
To me, it just looks like an empty function is passed to .then() callback, as if the snapshots hadn't been recorded at all...

simonsmith commented

Does it work correctly locally both in run and open mode? Do you run it locally in the same environment as CI (i.e Docker)?

awnclaim commented

Nope, locally it only runs well in Open mode, so the issue comes from the Run mode, it seems!

  • 👍1
stephsavoie commented

Same issue here. Additionnally, I realized that the snapshots that raise the same issue @awnclaim describes also throws the following error in the console:

Error: Image size (464x559) different than saved snapshot size (464x560).

In fact, it generates the diff image as well as a failure screenshot. But in my specific case, that issue seems to stop the test suite prematurely.

Other failing snapshots that are not throwing the error above had their diff image generated just fine.

Cypress 13.2.0
Vite 5.0.10
Node 20.8
Windows 10 Enterprise

simonsmith commented

It would be great if one of you could create a minimal reproducible example that I can debug. I'm using this plugin across four different projects with dozens of tests and I've not encountered. Perhaps there is something unique to your setup that I haven't seen, perhaps around filenames/paths

Thanks

  • 👀1