Concept/recipe for bigger data
hcomnetworkers opened this issue · 15 comments
Is it possible to load multiple objects at once? With bigger data and lazy-loading, subscribing to each individual object via an object/:id-like channel would cause too many db-requests - unless we'd debounce it server-side, which would cause additional lag.
Imagine I had to load the first 50 rows of a large table. This might be doable with one channel rows/ids to load all ids, and then another channel like { channel: 'rows/multi', ids: [1, 2, ..., 50] } for the first rows. But then what if the user scrolls down a bit and I need the next 20 rows? React would re-subscribe me with { channel: 'rows/multi', ids: [1, 2, ..., 70] }. This however seems very expensive - the first 50 rows would be loaded again.
Or do you see another way to not load things again upon a re-subscription of a bulk-load-action? The guide talks about the action.since.time, but the server is probably unaware of what data is already loaded.
Another issue is permissions on large data. As long as all users may see all the actions on a channel, things are fine. But what if a channel has to filter the result based on the user's permissions. How can I prevent the resend-actions from only reaching users allowed to see them?
For example, be there a channel openTerminations and user A gets [1, 2, 4], and user B gets [2, 3, 5] with sensitive data inside. User B edits termination 5 via an action and the server is configured to resend to channel openTerminations. How do I prevent user A from getting this action with possible sensitive data inside?
Our current project uses redux, downloads objects only once as long as they don't change and invalidates them via websocket-events. This worked great for us, but it has no offline-capability. That's why we are looking into logux, which seems very promising on this end!!
Good question. I like to think of how Logux architecture can fit different use cases. Feel free to ask more of these questions.
We definitely need a good article about it. Short answer:
This however seems very expensive - the first 50 rows would be loaded again.
You can check current client subscription on the server and avoid loading extra rows:
let rows = {}
server.channel('rows', {
async init (ctx, action) {
let prevIds = rows[ctx.clientId]
if (!prevIds) prevIds = rows[ctx.clientId] = []
let newIds = action.ids.filter(i => !prevIds.includes(i))
await Promise.all(newIds.map(async id => {
ctx.sendBack(await loadRow(id))
}))
prevIds.push(...newIds)
}
})
server.on('disconnected', client => {
if (client.clientId) {
delete rows[client.clientId]
}
})Another issue is permissions on large data.
filter callback in server.channel can help you here:
server.channel('rows', {
…,
async filter (ctx, action) {
let hasAccess = await loadUserRows(ctx.userId)
return (action, meta) => {
return hasAccess.includes(action.id)
}
}
})If we have an action with both sensitive and nonsensitive data, the solution is more tricky. You will need to create a new action with nonsensitive data only in server.on('add'.
server.on('add', (action, meta) => {
if (action.type === 'changePassword') { // this action contains password and can’t be re-send
server.log.add({ type: 'userChangedPassword', userId: action.userId }) // this action doesn’t have password
}
})Do my answers fit your use case?
Thank you, that already helps a lot!
Your guide mentions it tracks subscriptions - is it aware of additional data in the action, like the ids-array in the example here? So one component can load a different set of ids than another?
This hook automatically tracks all subscriptions and doesn’t subscribe to channel if another component already subscribed to the same channel.
Your guide mentions it tracks subscriptions - is it aware of additional data in the action, like the ids-array in the example here? So one component can load a different set of ids than another?
Sorry, right now it tracks subscription by JSON.stringify(action). If one component relies to { type: 'logux/subscribe', channel: 'rows', ids: [1, 2] } and other component relies on the same ids: [1, 2], useSubscription will call subscription only once. But if other relies on ids: [1, 2, 3], useSubscription will send { type: 'logux/subscribe', channel: 'rows', ids: [1, 2, 3] } to thre server again.
However, useSubscription is not so compilcated. You can write smarter useRows, which can generate logux/subscribe actions.
Okay, that's fine. Thanks so far!
I will keep it open until I will write good guides for this questions
server.channel('rows', { …, async filter (ctx, action) { let hasAccess = await loadUserRows(ctx.userId) return (action, meta) => { return hasAccess.includes(action.id) } } })
Hey, when is this outer filter executed? Every time an action is resent? Or just once upon subscribing? In the latter case: How can this hasAccess be invalidated if the permissions changed?
Furthermore: I think either your JSDoc or your guide is wrong. The JSDoc mentions that the returned filter-function also has a ctx as first argument, but it is missing in the guide and in this quote here.
when is this outer filter executed?
On every new action with this channel
How can this hasAccess be invalidated if the permissions changed?
Good question. You can load fresh data from database on every call.
Ok, I don't know I understand the filter.
The filter is supposed to filter to which clients an action is resend, is it not? I tried with a false response, but I still get resends on all clients:
server.channel<GroupsLoad, GroupsAll>('groups/load', {
async filter(ctx, action) {
return () => false;
},
});Good question. You can load fresh data from database on every call.
And no, the inner filter-function is not async, so I cannot reload from db on every call. But I suppose I can keep an updated synchronous cache outside the scope.
filter callback must be sync https://github.com/logux/server/blob/master/base-server.js#L917
We can add an async version too if you know the good use case for it.
https://github.com/logux/server/blob/master/base-server.js#L917 is the filter-callback on the channel, which can be async.
I think you mean https://github.com/logux/server/blob/master/base-server.js#L725
I am talking about filter creator. I think this could fix filtering:
server.channel<GroupsLoad, GroupsAll>('groups/load', {
- async filter(ctx, action) {
+ filter(ctx, action) {
return () => false;
},
});Ah, yes, and my Promise is thruthy, which is why the resends go through.
But why can't you change line 917:
let filter = i.filter && i.filter(ctx, action, meta)
to
let filter = i.filter && await i.filter(ctx, action, meta)
It's already an async function.
Of course, send PR to v0.6
Sorry, I am a noob with forking and pull requests.
It ended up in logux/server#68, too.
I am closing it right now. Task about the built-in solution is moved to logux/logux#11