Empire

A record store for Swift

Schema is defined by your types
Macro-based API that is both typesafe and low-overhead
Built for Swift 6
Support for CloudKit's CKRecord
Backed by a sorted-key index data store (LMDB)

Warning

This library is still pretty new and doesn't have great deal of real-world testing yet.

import Empire

@IndexKeyRecord("name")
struct Person {
    let name: String
    let age: Int
}

let store = try Store(path: "/path/to/store")

try await store.withTransaction { context in
    try context.insert(Person(name: "Korben", age: 45))
    try context.insert(Person(name: "Leeloo", age: 2000))
}

let records = try await store.withTransaction { context in
    try Person.select(in: context, limit: 1, name: .lessThan("Zorg"))
}

print(record.first!) // Person(name: "Leeloo", age: 2000)

Integration

dependencies: [
    .package(url: "https://github.com/mattmassicotte/Empire", branch: "main")
]

Data Modeling and Queries

Empire uses a data model that is extremely different from a traditional SQL-backed data store. It is pretty unforgiving and can be a challenge, even if you are familiar with it.

Conceptually, you can think of every record as being split into two tuples: the "index key" and "fields".

Keys

The index key is a critical component of your record. Queries are only possible on components of the index key.

@IndexKeyRecord("lastName", "firstName")
struct Person {
    let lastName: String
    let firstName: String
    let age: Int
}

The arguments to the @IndexKeyRecord macro define the properties that make up the index key. The Person records are sorted first by lastName, and then by firstName. The ordering of key components is very important. Only the last component of a query can be a non-equality comparison. If you want to look for a range of a key component, you must restrict all previous components.

// scan query on the first component
store.select(lastName: .greaterThan("Dallas"))

// constrain first component, scan query on the second
store.select(lastName: "Dallas", firstName: .lessThanOrEqual("Korben"))

// ERROR: an unsupported key arrangement
store.select(lastName: .lessThan("Zorg"), firstName: .lessThanOrEqual("Jean-Baptiste"))

The code generated for a @IndexKeyRecord type makes it a compile-time error to write invalid queries.

As a consequence of the limited query capability, you must model your data by starting with the queries you need to support. This can require denormalization, which may or may not be appropriate for your expected number of records.

Format

Your types are the schema. The type's data is serialized directly to a binary form using code generated by the macro. Making changes to your types will make deserialization of unmigrated data fail. All types that are stored in a record must conform to both the Serialization and Deserialization protocols. However, all index key members must also be sortable via direct binary comparison when serialized. This is not a property many types have, but can be expressed with a conformance to IndexKeyComparable.

Type	Key	Limitations
`String`	yes	none
`UInt`	yes	none
`Int`	yes	`(Int.min + 1)...(Int.max)`
`UUID`	yes	none
`Data`	yes	none
`Date`	yes	millisecond precision

`IndexKeyRecord` Conformance

The @IndexKeyRecord macro expands to a conformance to the IndexKeyRecord protocol. You can use this directly, but it isn't easy. You have to handle binary serialization and deserialization of all your fields. It's also critical that you version your type's serialization format.

@IndexKeyRecord("name")
struct Person {
    let name: String
    let age: Int
}

// Equivalent to this:
extension Person: IndexKeyRecord {
    public typealias IndexKey = Tuple<String, Int>
    public associatedtype Fields: Tuple<Int>

    public static var keyPrefix: Int {
        1
    }

    public static var fieldsVersion: Int {
        1
    }

    public var fieldsSerializedSize: Int {
        age.serializedSize
    }

    public var indexKey: IndexKey {
        Tuple(name)
    }

    public func serialize(into buffer: inout SerializationBuffer) {
        name.serialize(into: &buffer.keyBuffer)
        age.serialize(into: &buffer.valueBuffer)
    }

    public init(_ buffer: inout DeserializationBuffer) throws {
        self.name = try String(buffer: &buffer.keyBuffer)
        self.age = try UInt(buffer: &buffer.valueBuffer)
    }
}

extension Person {
    public static func select(in context: TransactionContext, limit: Int? = nil, name: ComparisonOperator<String>) throws -> [Self] {
        try context.select(query: Query(last: name, limit: limit))
    }
    public static func select(in context: TransactionContext, limit: Int? = nil, name: String) throws -> [Self] {
        try context.select(query: Query(last: .equals(name), limit: limit))
    }
}

`CloudKitRecord` Conformance

Empire supports CloudKit's CKRecord type via the CloudKitRecord macro. You can also use the associated protocol independently.

@CloudKitRecord
struct Person {
    let name: String
    let age: Int
}

// Equivalent to this:
extension Person: CloudKitRecord {
    public init(ckRecord: CKRecord) throws {
        try ckRecord.validateRecordType(Self.ckRecordType)

        self.name = try ckRecord.getTypedValue(for: "name")
        self.age = try ckRecord.getTypedValue(for: "age")
    }

    public func ckRecord(with recordId: CKRecord.ID) -> CKRecord {
        let record = CKRecord(recordType: Self.ckRecordType, recordID: recordId)

        record["name"] = name
        record["age"] = age

        return record
    }
}

Optionally, you can override ckRecordType to customize the name of the CloudKit record used. If your type also uses IndexKeyRecord, you get access to:

func ckRecord(in zoneId: CKRecordZone.ID)

Questions

Why does this exist?

I'm not sure! I haven't used CoreData or SwiftData too much. But I have used the distributed database Cassandra quite a lot and DynamoDB a bit. Then one day I discovered LMDB. Its data model is quite similar to Cassandra and I got interested in playing around with it. This just kinda materialized from those experiments.

Can I use this?

Sure!

Should I use this?

User data is important. This library has a bunch of tests, but it has no real-world testing. I plan on using this myself, but even I haven't gotten to that yet. It should be considered functional, but experimental.

Contributing and Collaboration

I would love to hear from you! Issues or pull requests work great. Both a Matrix space and Discord are available for live help, but I have a strong bias towards answering in the form of documentation. You can also find me on mastodon.

I prefer collaboration, and would love to find ways to work together if you have a similar project.

I prefer indentation with tabs for improved accessibility. But, I'd rather you use the system you want and make a PR than hesitate because of whitespace.

By participating in this project you agree to abide by the Contributor Code of Conduct.

mattmassicotte/Empire