Enable multiple sinks per table/collection by adding additional conditions on invalidation
Closed this issue · 1 comments
Is your feature request related to a problem? Please describe.
At the moment, it's not possible to write to the same table using multiple sinks. Insertion works fine, but during invalidation one indexer deletes the other indexer's data too.
Describe the solution you'd like
I believe the easiest solution is to add a new option with a list of additional conditions for data invalidation.
For example:
export const config = {
sinkOptions: {
invalidate: [
{ column: 'my_column', value: 'abcd' },
{ column: 'other_column', value: 'aaa' }
]
}
}
Obviously the user script must return objects that have my_column
and other_column
as attributes, even tho this behavior won't be enforced for simplicity sake.
All fields in invalidate
will be added as AND
expressions to the invalidate query, the condition being <column>==<value>
.
Describe alternatives you've considered
We can add a callback in the script file to return the conditions at runtime. I believe it adds additional complexity for no reason.
Additional context
Affects the postgres and mongo sinks.
This is now available.