pointfreeco/swift-snapshot-testing

Running tests with Xcode 14.1 changes frame?

chriisong opened this issue · 4 comments

chriisong commented

SnapshotTesting uses GitHub issues for bugs. For more general discussion and help, please use GitHub Discussions.

I've been using 13.4 for a while now, just tried running our app with 14.1 and saw that all my recursiveDescription assertions failed. Is this what everyone experienced with upgrading to Xccode 14.x? I didn't change anything else on my end except to open my project with Xcode 14.1

mathaeus commented

We also experience failures migrating from Xcode 13 to Xcode 14. For most cases there is no visible difference yet still the majority of tests fail.
It's not the first time that after an Xcode update some snapshots failed, but it's becoming more and more of a time killer for us :/

So I'd be also interested how others approach this.

chriisong commented

actually, i see some 2px frame changes in Xcode 14 🤔

but to your point @mathaeus , I also wonder if users just re-record the changes that occurred as a result of Xcode major version upgrade (given that there are no visible or very little visible changes)

mathaeus commented

What I ended up doing was:

  • Update the device that we were testing against (i.e. re-recording snapshots)
  • using the perceptual precision
  • 👍1
dafurman commented

I also wonder if users just re-record the changes that occurred as a result of Xcode major version upgrade (given that there are no visible or very little visible changes)

@chriisong Generally, yes, this is what we do at my team that uses https://github.com/uber/ios-snapshot-test-case which suffers the same issue. Differences between Xcode versions can subtly influence the way views are laid out.

Oftentimes, we end up seeing this as text characters being shifted slightly by a single pixel.
If you don't want to loosen your pixel precision, depending on how many failures you've got, the simplest way to resolve this is generally to rerecord your tests, and then do a comparison of the before / after images to ensure nothing looks too different.

  • 👍1