Add "entity" mode to Postgres and MongoDB sinks
Closed this issue · 0 comments
Is your feature request related to a problem? Please describe.
The current sinks stores all values as a logical list (e.g. a list of transfers, a list of player moves, etc). The issue arises if I need to know the latest account balance, the player status, etc. since I need to, for each entity, only fetch the latest value.
Describe the solution you'd like
We should add an "entity mode" where the sink updates the rows in the table as if they represent an entity.
The user opts-in this feature by setting the entity
property on the sink:
export const config = {
sinkOptions = {
// enable entity mode
entity: {
keys: ['id', 'realm']
}
}
}
Then the behavior of the indexer changes as follows:
-
_cursor
becomes a range (int8range
in postgres,{from: number, to: number}
in mongo). - handle_data:
- collect keys for returned rows
- update: set upper bound on
_cursor
toendCursor
for all rows where id in keys and_cursor.to
=$\infty$ . - insert: rows returned by handle_data
- invalidate:
- delete all rows where
_cursor.from
> invalidated cursor (same as now) - update all rows where
_cursor.to > invalidated
: set_cursor.to
to$\infty$ .
- delete all rows where
Users can then query the most recent data by selecting rows where _cursor.to
=
Additional context
We need this for postgres and mongo sinks.