Eager loading of sibling relation causes fatal error
karola-gr opened this issue · 30 comments
Description
Eager loading of sibling relation causes fatal error.
Fatal error: Cannot access field before it is initialized or fetched
The data in the database seems to be correctly created. The relations exist and are filled. However, I cannot access them in a query. The problem only occurs with some sibling relations, but only when querying a sibling that is held by an optional parent, or with a sibling that may be empty. Not with all of them.
Details
Case 1:
Sibling relation of optional parent
Example EntityOne with sibling relation to EntityTwo
final class EntityOne: Model, Content {
static let schema = "entity_one"
@Siblings(
through: EntityOneAttachedEntityTwo.self,
from: \.$entityOne,
to: \.$entityTwo
) var attachedEntityTwo: [EntityTwo]
init() {}
EntityThree with optional parent relation to EntityOne
final class EntityThree: Model, Content {
static let schema = "entity_three"
@ID var id: UUID?
...Fields..
@OptionalParent(key: "entity_one_id") var entityOne: EntityOne?
init() {}
}
--> Query
let entity = try await EntityThree
.query(on: req.db)
.with(\.$entityThree) { entityThree in
entityThree
.with(\.$field) // Fields, Parents, etc. work fine
.with(\.$attachedEntityTwo) // Causes fatal error in all()
}
.all()
}
BUT: entity.entityThree.load(on: db) works fine
Case 2:
Same behaviour as Case 1, but without optional parent relation
Example EntityOne with sibling relation to EntityTwo
final class EntityOne: Model, Content {
...
@Siblings(
through: EntityOneAllowedEntityTwo.self,
from: \.$entityOne,
to: \.$entityTwo
) var allowedEntityTwo: [EntityTwo]
....
}
RelationTable (same for case 1)
final class EntityOneAllowedEntityTwo: Model {
...
@ID(key: .id) var id: UUID?
@Parent(key: "entity_one_id") var entityOne: EntityOne
@Parent(key: "entity_two_id") var entityTwo: EntityTwo
init() {}
init(id: UUID? = nil,
entityOne: EntityOne,
entityTwo: EntityTwo
) throws {
self.id = id
self.$entityOne = try entityOne.requireID()
self.$entityTwo = try entityTwo.requireID()
}
}
Eager loading in query failes, but loading afterwards works fine.
let entityOne = try await EntityOne
.query(on: db)
.with(\.$field) // Everything works fine
// .with(\.$allowedEntityTwo) // Causes fatal error
.first()
.unwrap(or: Abort(.notFound))
.get()
But: try await entityOne.$allowedEntityTwo.loadIfNeeded(on: db) also works fine.
Migration for relation table
struct CreateEntityOneAllowedEntityTwoPivot: Migration {
func prepare(on database: Database) -> EventLoopFuture<Void> {
database.schema(EntityOneAllowedEntityTwo.schema)
.id()
.field("entity_one_id",
.uuid,
.references(EntityOne.schema, "id", onDelete: .cascade),
.required)
.field("entity_two_id",
.uuid,
.references(EntityTwo.schema, "id", onDelete: .cascade),
.required)
.unique(on: "entity_one_id", "entit_two_id")
.create()
}
func revert(on database: Database) -> EventLoopFuture<Void> {
database.schema(EntityOneAllowedEntityTwo.schema)
.delete()
}
}
Expected behavior
Sibling relations should be eager loaded / accessed in query
Environment
Vapor 4.77
MacOS: 13.5
maybe need use .with(.$field.$pivots)
We tested it, but unfortunately the error persists.
What does your code look like that actually triggers the error (where you're accessing the properties)
The error comes from the query, as I said, when loading the sibling relation.
The code for this is basically this:
func getAllHandler(
_ req: Request
) async throws -> [EntityOneReturnDto] {
return try await EntityOne
.query(on: req.db)
.with(.$allowedEntityTwo) <- Sibling relation (.$pivots) leads to the same issue
...Fields....
.all()
}
The entity and the migration look as previously described..
I attach the stack trace as a screenshot. Maybe that helps.
In the code snippet you're returning a DTO - are you sure it's not happening there?
yes, I'm sure. The error is thrown before the dto is created and then of course when I remove the dto completely from the code. I think it fails in the call to the all() function in the query. At least that's the last point I can reasonably track with the debugger.
Do you have any idea where else this could be coming from, or can I provide any other information to look for the problem?
Sorry to ask again...but we are kind of stuck here
@karola-gr I've been unable to reproduce this. Do you have a reproducible project you can share?
@karola-gr does it give fatalError if you don't select specific fields from the table?
@Kishimotovn No, this only happens with the specific sibling relations, not with other fields. as soon as I exclude the siblings from the query and reload them directly afterwards, it works without problems. Maybe there is some problem with the combination with the optionals that something is assumed to be nil... But yes...reload the relation is also our current workaround.
@0xTim , it is difficult to recreate the scenario in an minimal example project. I think the only way would be to look at our project remotely in a call or something.
Hi :) I have set up a small gitlab project in which I have rebuilt the part of the original project that fails. Maybe this will help. @0xTim ...I have invited you to the repo.
Hello,
I wanted to follow up on the repository invitation I sent to reproduce the problem we discussed previously. Have you had an opportunity to review it? Are there any updates or feedback on the issue?
I appreciate your time and assistance. Thank you.
@karola-gr I've just tried running the tests from the GH repo you uploaded and I don't get any crashes. It retrieves items from the DB and returns them and doesn't fail like you are seeing. I get
Test Suite 'Selected tests' started at 2023-11-07 14:06:07.143.
Test Suite 'AppTests.xctest' started at 2023-11-07 14:06:07.143.
Test Suite 'Get' started at 2023-11-07 14:06:07.143.
Test Case '-[AppTests.Get testGetExercise]' started.
[GET /api/exercises, POST /api/exercises, GET /api/exercise_bases, POST /api/exercise_bases, POST /api/exercise_standards, POST /api/exercise_variation_categories]
2023-11-07T14:06:07+0000 info codes.vapor.request : request-id=43FC3D0D-E803-4010-AC7B-81F4CF64282B [Vapor] GET /api/exercises
DB8CDEA1-89BE-468E-B482-02B51704A406 Bench Press
CED1F8E8-9DD3-4FCA-B160-17DD540AD133 Bench Press
/Users/timc/Developer/playgrounds/vaportestenv/Tests/AppTests/Get.swift:36: error: -[AppTests.Get testGetExercise] : XCTAssertEqual failed: ("2") is not equal to ("0")
Test Case '-[AppTests.Get testGetExercise]' failed (15.481 seconds).
Test Suite 'Get' failed at 2023-11-07 14:06:22.625.
Executed 1 test, with 1 failure (0 unexpected) in 15.481 (15.481) seconds
Test Suite 'AppTests.xctest' failed at 2023-11-07 14:06:22.625.
Executed 1 test, with 1 failure (0 unexpected) in 15.481 (15.482) seconds
Test Suite 'Selected tests' failed at 2023-11-07 14:06:22.625.
Executed 1 test, with 1 failure (0 unexpected) in 15.481 (15.482) seconds
Program ended with exit code: 1
Are you running the latest packages etc?
Yes, I updated all the packages again, but still got this:
Test Suite 'Selected tests' started at 2023-11-07 18:32:08.480.
Test Suite 'AppTests.xctest' started at 2023-11-07 18:32:08.481.
Test Suite 'Get' started at 2023-11-07 18:32:08.481.
Test Case '-[AppTests.Get testGetExercise]' started.
[GET /api/exercises, POST /api/exercises, GET /api/exercise_bases, POST /api/exercise_bases, POST /api/exercise_standards, POST /api/exercise_variation_categories]
2023-11-07T18:32:08+0100 info codes.vapor.request : request-id=474AE927-1FD9-4FEC-B7AC-A65C52322BCD [Vapor] GET /api/exercises
FluentKit/Field.swift:23: Fatal error: Cannot access field before it is initialized or fetched: exercise_base_id
2023-11-07 18:32:13.697927+0100 xctest[62542:10452040] FluentKit/Field.swift:23: Fatal error: Cannot access field before it is initialized or fetched: exercise_base_id
Well, this confuses me even more now :D
The best thing would be do a clean repo clone and follow the steps. If that fails then you must have a code difference. You can see the line in the stack trace that causes the issue
So I tried a few things again today. I used the postgres and the redis. I deleted derived data, did a clean build, recloned the repo in exactly the same state, reloaded all packages...
(M2, Xcode 15.0.1, macOS14)
I always did:
TestExerciseVariationCategory
TestExerciseBase
TestExerciseStandard
Test Exercise
so uploads are in the right order, then TestGetExercise and the error occurs.
In a few cases, which were not reproducible for me, it worked. The first time was when I switched to linux..then I tried to build the original project on linux and that did not work. The test repo didn't work after that either...it went on like that all day. It feels really random... :(
The stack trace doesn't really get me anywhere either, to be honest. It breaks between the .all() and the dto
Oh I think I know what's going on. The error message is actually subtly different and it's nothing to do with sibling relations.
The error is about specific fields and not relations - I'm guessing you have a model field that has a null column in the database that should be @OptionalField and not @Field - that would explain the behaviour you're seeing
i created the data in such a way that there are two entries that are null in one column each (exerciseBaseId in the first entry andexerciseStandardId in the other). However, these two columns are declared as optionalParents. Even if I replace all null entries with data, I still get the error.
Reloading these fields also works without any problems. But it does not work directly in the query.
I am trying to access its siblings in the query via an optional parent. Maybe it searches for null in the sibling relation when the optional parent is empty and crashes at this point?
I removed the optional parent again and only looked at the sibling relation. Here I get a nil when it crashes, but there shouldn't actually be a nil. It is required and I don't see any entry in the database that has a nil in this column.
So yes, your previous answer is probably a problem, but how can this happen? Especially since this case does not occur when reloading after the query
I have double checked the data on the database as well as the db dump and it seems all correct. Also a nil value in a required column should already lead to a problem when saving to the database or lazy load them, right?
If I check the ids in
private struct SiblingsEagerLoader<From, To, Through>: EagerLoader
these are also correct and not nil. I don't get the point where the problem occurs :(