3-layer composition?

[jbmaxwell]

I'm trying to build a 3-layer, memory+disk+server (Firebase), composition, but I'm getting a "Type of expression is ambiguous without more context" error on the first line.

    let myObjectMemoryCache = MemoryStorage<Filename, MyObject>()   // <-- ambiguous type error
        .combined(with: DiskStorage.main.folder("myObjects", in: .documentDirectory).mapJSONObject(MyObject.self))
        .combined(with: RemoteStorage.main.collection("myObjects").mapJSONObject(MyObject.self))

I'll admit, I'm really not sure how I'm supposed to do this. It would be great to have a little more info on how these kinds of custom compositions should be implemented. (I should mention that I do have a running memory+disk version that works fine, based on the examples you provide—I'm just not sure how to add another layer.)

[dreymonde]

Hi! Will answer your question in about twenty minutes 🙂

[dreymonde]

So
I agree that this is not explained well enough, my bad
I guess that the issue here is Keys type on your RemoteStorage, I suppose it doesn’t use the same Filename type as DiskStorage and MemoryStorage<Filename, MyObject>. If that’s the case, the solution would be to ditch Filename and just add .usingStringKeys() to DiskStorage, which will convert its keys from Filename to String.

[dreymonde]

Also keep in mind that this kind of 3-layer composition will have its issues, mostly about writes. So if say you’ll want you have a three-layer Storage<String, String>:

1. Memory
2. Disk
3. Firebase

Let’s say that you want to:

composedStorage.set("new-value", forKey: "key-1")

Your memory and disk storages will almost always succeed, but network storage can fail. Now you have inconsistent storage: memory and disk have «new-value», while Firebase still has some let’s call it «old-value».

And when you’ll try to retrieve the data like this:

composedStorage.retrieve(forKey: "key-1") { … }

You will get a data from your memory storage directly (or disk storage if your app is restarted), so you’ll get a «new-value»… but Firebase, which should be the real source of truth, still has «old-value». That’s definitely a bug, and you will need to build a whole system around it.

That’s why I’m not generally recommending to use Shallows in such a manner. Network-based storages are fine as long as they’re read-only. If you want to support writes as well, you’ll be better with building some custom solution.

[jbmaxwell]

First, thanks for the incredibly speedy response!

I see what you mean about having a kind of hidden implementation of the remote storage, buried inside Shallows—could make things trickier than need be. So I suppose perhaps a better solution would be to have some class/service dedicated to keeping the DiskStorage and Firebase in sync? Just out of curiosity, is there any built-in notification for responding to changes in DiskStorage?

[dreymonde]

Yes having this kind of service is a good idea. There is no such notifications built-in, but you can easily add such functionality in extensions on Storage.

[jbmaxwell]

Perfect, thanks!

[dreymonde]

Here’s an example of what I mean by that. It uses Alba as a reactive solution but you can easily adapt it to use anything you want, including NSNotifications.

import Alba
import Shallows

public struct ReportingStorage<Key, Value> : StorageProtocol {
    
    private let underlying: Storage<Key, Value>
    
    public let didRetrieve = Publisher<Result<Value>>(label: "ReportingStorage.didRetrieve")
    public let didSet = Publisher<Result<Void>>(label: "ReportingStorage.didSetWithResult")
    
    init(_ storage: Storage<Key, Value>) {
        self.underlying = storage
    }
    
    public func retrieve(forKey key: Key, completion: @escaping (Result<Value>) -> ()) {
        underlying.retrieve(forKey: key, completion: { result in
            completion(result)
            self.didRetrieve.publish(result)
        })
    }
    
    public func set(_ value: Value, forKey key: Key, completion: @escaping (Result<Void>) -> ()) {
        underlying.set(value, forKey: key) { (result) in
            completion(result)
            self.didSet.publish(result)
        }
    }
    
}

extension StorageProtocol {
    
    public func reporting() -> ReportingStorage<Key, Value> {
        return ReportingStorage(asStorage())
    }
    
}

And then you can just:

let disk = DiskStorage.main.folder("cache", in: .cachesDirectory)
let diskReporting = disk.reporting()
diskReporting.didSet.proxy.listen { (result) in
    print(result)
}

[jbmaxwell]

This is great, thanks!

[jbmaxwell]

I'm having trouble getting the reactive solution above working. I think (maybe) it's because I'm using a combined memory+disk storage(??). I've put a break in ReportingStorage .set(), but it never gets hit (though a break in .set() in my memoryCache does get hit). My basic setup uses storageResponder = memoryCache.reporting() to set up the listener. Any thoughts as to what might be going on?

[dreymonde]

Hi! I’ll be able to inspect your question more in-depth tomorrow. Meanwhile, could you please provide some more code? Ideally your composition setup code (creating memory+disk reporting storage) and usage code (the place where you use the mentioned storage).

[jbmaxwell]

Great, thanks.
My "ReactiveStorage" is basically identical to your ReportingStorage above.

public class LocalStorageManager {

    public let storageResponder: ReactiveStorage<Filename, MyObj>

    // the memoryCache is a property of the service class "LocalStorageManager"
    let memoryCache = MemoryStorage<Filename, MyObj>()
        .combined(with: DiskStorage.main.folder("myobjs", in: .documentDirectory).mapJSONObject(MyObj.self))

    // ...other stuff
    // ...

    init() {
        // creating LocalStorageManager sets up the storageResponder (reporting storage)
        storageResponder = memoryCache.reporting()
        storageResponder.didSet.proxy.listen { (result) in
            print("MyObj set with result: \(result)")
        }
    }

}

[dreymonde]

Hm, I can’t see the reason .set() on the reactive storage is not being hit.
To be clear — when retrieving/setting data on a storage, you need to use your storageResponder object instead of memoryCache. I suppose this is the problem.

[jbmaxwell]

Ah, of course... yes, my previous code was calling set on memoryCache, not storageResponder! Will update that now. Thanks.

[dreymonde]

Great! Let me know if you have any problems.