swiftlang/swift-testing

API to reliably detect if a test is running under Swift Testing or XCTest

Opened this issue · 4 comments

younata commented

Description

Like #474, this was also discussed in a lab.

This issue is related to the second half of Quick/Nimble#1145: Determining whether to report errors to Swift Testing or XCTest.

Background about how Nimble determines the test runner

Nimble nominally supports working with other test runners, so that if you wanted, you could use non-XCTest test runners.

The way Nimble works is it dynamically determines the test runner environment the first time an assertion fails. At that point, Nimble checks whether the test runner is XCTest, and then reports the issue to XCTest using the "appropriate" API1. Then, Nimble will cache that check.

The way Nimble checks if the test runner is XCTest is by checking if the XCTestCase class can be found. Which probably needs to be updated. For example, I would expect that check to still pass if Swift Testing is the test runner, just because I would also expect XCTest to be loaded in to the test bundle even if it's not actually used. Especially during this period when Swift Testing is just being introduced, and I can expect a lot of wonkiness going on. So, I'll probably definitely update that check to be more like "is there a currently running XCTestCase"?

The actual ask here

In the lab I had, it was mentioned that the only API for determining if a Swift Test is running is Test.current. However, as you mentioned, Test.current is a task local value, so if someone uses a detached Task inside of the test, then Test.current will incorrectly return nil (as far as the semantics of checking "am I running inside of a test?" go).

What Nimble needs here is a way to reliably and positively identify whether or not it should report the issue to Swift Testing.

Other Possible Approaches for reporting issues (Nimble Specific)

Also in Quick/Nimble#1145, I wrote up that another way that Nimble could determine whether or not to report to Swift Testing or XCTest is to assume that if there is no current running XCTestCase, then it should fall back to Swift Testing. Which probably would work in the interim, but I'd certainly appreciate a way to positively verify that Nimble should report the failure to Swift Testing.

Another approach I wrote up is to try to report to both Swift Testing and XCTest. So, just blasting that a test failure happened (or an error was encountered, or a polling expectation timed out, etc.) to both of them, and hoping that only one of them records the failure. That's also unappealing to me because, again, I'd like to positively verify the right place to report the issue and only report it once. But it's certainly an approach.

Gosh, Rachel, that's a lot. Is there a TL;DR?

I need a way to reliably and positively identify if Nimble (and other testing assertion tools) should report issues to Swift Testing or XCTest.

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

swift-testing version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

No response

Footnotes

  1. Which just inspired me to write up https://github.com/Quick/Nimble/issues/1146, because Nimble should be using XCTestCase.recordFailure. ↩

grynspan commented

Hard part: Swift Testing and XCTest don't know about each other, so it's not clear how to write a "one true test library" function or something like that.

You can check if Test.current is nil or not, which will tell you if the current task is running a Swift Testing test. That doesn't really help you with code that's running on a detached task or background thread.

I could see us adding a Test.isRunning global boolean property that just tracks if any Runner is currently running. It can be implemented trivially with:

// increment when runner starts, decrement when it ends
private static let _isRunning = Locked(rawValue: 0)

public static var isRunning: Bool {
  _isRunning.rawValue > 0
}

That should be sufficient API to tell you if Swift Testing is running; if it's not, you could then assume XCTest is running. Would that work?

stephencelis commented

@grynspan This would definitely be nice to have! Our test tools are currently having to resort to inspecting the process info to determine if the current process is running tests, which is failure-prone.

It'd be nice for XCTest to also expose such an API, but I'm guessing that won't happen at this point 😉

younata commented

That should be sufficient API to tell you if Swift Testing is running; if it's not, you could then assume XCTest is running. Would that work?

That would work well enough for now. The next question is being able to record issues without crashing. Last I checked, swift testing throws a runtime error if you try to record an issue when Test.current is nil. As I recall in our lab discussion, someone brought up probabilisticly attempting to determine the test that the error is a part of? Even in the interim, recording that an issue happened that couldn't be linked to a test is better than silently dropping it.

grynspan commented

Last I checked, swift testing throws a runtime error if you try to record an issue when Test.current is nil.

This is a known bug in Xcode 16 Beta 1 and is not a bug in Swift Testing (which simply records the issue with no associated test.)

  • 👍1