apibara/dna

Add "entity" mode to Postgres and MongoDB sinks

Closed this issue · 0 comments

fracek commented

Is your feature request related to a problem? Please describe.

The current sinks stores all values as a logical list (e.g. a list of transfers, a list of player moves, etc). The issue arises if I need to know the latest account balance, the player status, etc. since I need to, for each entity, only fetch the latest value.

Describe the solution you'd like

We should add an "entity mode" where the sink updates the rows in the table as if they represent an entity.
The user opts-in this feature by setting the entity property on the sink:

export const config = {
  sinkOptions = {
    // enable entity mode
    entity: {
      keys: ['id', 'realm']
    }
  }
}

Then the behavior of the indexer changes as follows:

  • _cursor becomes a range (int8range in postgres, {from: number, to: number} in mongo).
  • handle_data:
    • collect keys for returned rows
    • update: set upper bound on _cursor to endCursor for all rows where id in keys and _cursor.to = $\infty$.
    • insert: rows returned by handle_data
  • invalidate:
    • delete all rows where _cursor.from > invalidated cursor (same as now)
    • update all rows where _cursor.to > invalidated: set _cursor.to to $\infty$.

Users can then query the most recent data by selecting rows where _cursor.to = $\infty$.

Additional context

We need this for postgres and mongo sinks.