Snapshot counter breaks test repetitions (inc. retry on failure)
lukeredpath opened this issue · 1 comments
Describe the bug
There are some scenarios in which a test case can be run multiple times within the same XCTest process - once is when you have retry on failure configured in your test plan, the other is when using the "Run Repeatedly" function in Xcode.
By default, snapshot names take the form testName.<counter> where counter is an incrementing number starting from 1 - this is what allows multiple snapshots in a single test without explicit names. Unfortunately, because this counter is not reset between iterations, subsequent iterations will fail as it will look for the wrong snapshot name.
To Reproduce
Create any test case with a single snapshot and no explicit name - use Xcode's "Run Repeatedly" to run the test more than once without relaunching between tests and the test will fail.
Expected behavior
The test should not fail.
Additional context
Ideally this counter would be reset between test iterations - I don't know if there is an API to check which iteration a test is on. If there is no automatic solution to this, an API to reset the counter that could be called from setup() would at least allow the counter to be reset explicitly between each test iteration.
It might be possible to use the XCTestObservationCenter to install an observer that resets this counter between test iterations.
Snapshots with explicit names (using the named: parameter) do not have this issue.
Oh, looks like this was fixed in #585 - I tried searching but couldn't find this.