pointfreeco/swift-snapshot-testing

Xcode Cloud: Snapshot references don't match, visually the same. Locally passing

Alvin-Robert-Lee opened this issue · 6 comments

Alvin-Robert-Lee commented

Has anyone experienced snapshots failing to match their reference on Xcode Cloud even though they match locally?

After fixing the issue where snapshots weren't being copied over, I still have some snapshots failing to match their reference image snapshot, even though they are the same visually. I've tried lowering the precision down to 0.98 and it still fails.

I've confirmed that the simulator iPhone and iOS version match my local environment. Xcode version also matches.

The only difference I could find is that the macOS version on Xcode Cloud is 13.4.1, and I'm on 13.4.

Alvin-Robert-Lee commented

Seems like Xcode Cloud runners have Intel CPUs, which cause snapshots to diff+fail if you generated your snapshots on a machine running Apple Silicon.

krissaj commented

how did you resolve this?

fespinoza commented

I tend to use the precision and perceptualPrecision parameters set to 0.98. This allows us for some "wiggle room" in the snapshots and works ok

krissaj commented

Thank you very much, i will try that! but did you have any other issue of accessing the reference images in Snapshot folder? For me, all tests pass locally but in xcode cloud all of them fails and there is no reason in logs.

fespinoza commented

I made a YouTube video exactly about that! https://www.youtube.com/watch?v=KPJL8vN3lAg

tdrhq commented

@krissaj @fespinoza

The solution is really to just record on the CI. That way snapshots are always being generated on identical devices. (You can still test locally against locally recorded snapshots.)

But you don't have to actually save the snapshots into the repository, take a look at https://screenshotbot.io (also open source): our plugin will handle all of this for you and you'll get notifications on your Pull Request when snapshots change, so you don't have to manually record snapshots each time.

We're also actively being used by large Android and iOS teams, it's especially useful when you want to scale your snapshots to 100s or 1000s of snapshots and don't want to force developers to manually re-record screenshots.