Crashes during unit test execution

[nashysolutions]

Describe the bug

• Crashes occur in testing, within swift-nio and async-kit.
• The bug has been raised here.

To Reproduce

Run tests.

All of my tests subclass a base class. Initially that base class looked like this.

class BaseTestCase: XCTestCase {
    
    let application = Application(.testing)
    
    override func setUp() async throws {
        try await configure(application)
        continueAfterFailure = false
    }
    
    override func tearDown() async throws {
        application.shutdown()
    }

But I realised that this resulted in the application instance being used by the subsequent test after it was shutdown in the tear down of the previously running test.

So I modified to this.

class BaseTestCase: XCTestCase {
    
    var application: Application!
    
    override func setUp() async throws {
        let application = Application(.testing)
        try await configure(application)
        self.application = application
        continueAfterFailure = false
    }
    
    override func tearDown() async throws {
        application.shutdown()
    }

    ...

But now I hit a precondition where deinit happens before shutdown.

Expected behavior

The tests run.

Environment

• Vapor Framework version: 4.85.1
• Vapor Toolbox version: ?
• Xcode: Version 15.0.1 (15A507)
• Swift: swift-driver version: 1.87.1 Apple Swift version 5.9 (swiftlang-5.9.0.128.108 clang-1500.0.40.1) Target: arm64-apple-macosx14.0
• OS version: 14.1 (23B74)

[nashysolutions]

The async setup func may not be on main thread. Would that be an issue? Will check that later.

[0xTim]

@nashysolutions so that normally means something threw an error during start up (try wrapping the configure in a do/catch), causing the app to not spin up properly so that when it comes to teardown it's not in the right state. Are there are errors in the logs?

[0xTim]

(If that doesn't provide any pointers is it possible to see the test code? Something weird is going on. Is it always the same test the crashes or any test?)

[nashysolutions]

If I run all of the tests, the failure happens on a specific test.

2023-11-10T04:50:00+0000 info codes.vapor.request : request-id=0301B138-DB3B-453B-9DE6-71F937CC7C32 [Vapor] POST /users
ERROR: Cannot schedule tasks on an EventLoop that has already shut down. This will be upgraded to a forced crash in future SwiftNIO versions.
ERROR: Cannot schedule tasks on an EventLoop that has already shut down. This will be upgraded to a forced crash in future SwiftNIO versions.
BUG in SwiftNIO (please report), unleakable promise leaked.:480: Fatal error: leaking promise created at (file: "BUG in SwiftNIO (please report), unleakable promise leaked.", line: 480)
2023-11-10 04:50:00.698983+0000 xctest[39434:1275754] BUG in SwiftNIO (please report), unleakable promise leaked.:480: Fatal error: leaking promise created at (file: "BUG in SwiftNIO (please report), unleakable promise leaked.", line: 480)

If I run that specific test in isolation, it runs to completion without error.

[0xTim]

@nashysolutions which project and which test in that project is it?

[nashysolutions]

I've found the issue.

The aforementioned test had a logical assert condition which was not met by my business logic. A tear down block wasn't triggered afterwards, due to continueAfterFailure being set to false.

This lead to an EventLoopFuture / EventLoopPromise imbalance that is quite aggressively enforced in debug builds.

/*! - XCTestCase
 * @property continueAfterFailure
 * Determines whether the test method continues execution after an XCTAssert fails.
 *
 * By default, this property is YES, meaning the test method will complete regardless of how many
 * XCTAssert failures occur. Setting this to NO causes the test method to end execution immediately
 * after the first failure occurs, but does not affect remaining test methods in the suite.
 *
 * If XCTAssert failures in the test method indicate problems with state or determinism, additional
 * failures may be not be helpful information. Setting `continueAfterFailure` to NO can reduce the
 * noise in the test report for these kinds of tests.
 */
@property BOOL continueAfterFailure;

XCTest runs the teardown methods once after each test method completes: first tearDown(), then tearDownWithError(), then tearDown() async throws. Avoid preparing state for subsequent tests in the teardown methods. XCTest doesn’t guarantee that it will call teardown methods

There's no obvious fix, so perhaps updating the docs to say you guys don't support continueAfterFailure will suffice?

Or perhaps shutdown should also be called in deinit of the test case?

[gwynne]

continueAfterFailure is badly broken on Linux in any event; it's never been safe to use it.

[nashysolutions]

FYI: adding shutdown in deinit of the test case did not resolve the issue. Will avoid continueAfterFailure going forward.