Any way to update resgate collections in bulk?

[ellis]

I'm testing resgate with a large database and need to transfer about 1000 items from a database. Scanning the database's index takes about .25 seconds, whether it returns 1 or 1000 items. The way that resgate is set up appears to require fetching each item individually, 1000 times, which leads to excessively long response times. Is there any way to optimize this, besides using methods (which won't automatically update on the client)?

[jirenius]

At the moment, there is no "bulk" support, neither for get requests (each resource is fetched separately), nor for events (each add/remove event is sent separately. But large changes in a collection can be handled with a system.reset event instead).

Service-side caching

The common way this is solved is by service-side caching: When a collection of items/models is fetched from the database, the service will cache the models for a while in a simple key-value cache.

Preload feature?

One idea (the was scrapped) was to add some preload feature, similar to HTTP2's preload.

Example:

get.my.collection

Response (not valid! Do not use):

{
   "result": {
      "collection": [ { "rid": "my.model.1" }, { "rid": "my.model.2" } ], // The collection
      "preload": { //  "Preloaded" models that Resgate otherwise would fetch
         "models": { 
            "my.model.1": { "name": "foo" },
            "my.model.2": { "name": "bar" }
         }
      }
   }
}

The caveats is that, Resgate would need to start subscribing to event.my.model.1.* (and 2) prior to receiving the data, or else it might miss some change event and end up with old state. But it won't know which resources it needs to subscribe to until after receiving the response.

This is a common problem with real-time data.

I am very open for suggestions on how to improve this though! :)

[jeteon]

What about having a "model" that is a collection scoped to your needs? So, get.item.collection returning something like:

{
   "items": [
        {},
        {}
   ]
}

It sounds like those 1000 items are effectively just similar pieces of some meaningful resource. It might not be such an offence to the design.

[ellis]

Ok, thanks for the input and ideas, guys.