Which items should write to a database?

[spine-o-bot]

In GitLab by @soininen on Nov 12, 2020, 11:16

To transform data from one database to another, users currently need to build the following DAG:

Data Store 1 -> Data Transformer -> Combiner -> Data Store 2

Having to insert a Combiner between Data Transformer and Data Store 2 is counterintuitive.

A few options to ease the situation were discussed on Slack:

1. Data transformer could write to downstream Data Stores removing the need to the Combiner item above.

• This would essentially make Data Transformer do Combiner's job so we could consider dropping Combiner item
• We'd need to add extra writing options to Data Transformer such as 'purge before write' and rules how to deal with conflicting items in all inpu databases
• Data transformers name does not really match its usage to combine databases

2. Consider renaming Combiner to e.g. Writer as it can write a single database just fine.

• Item names would match their usage
• Extra database write options would be in a dedicated project item

3. Make Data Store write to itself if it receives a database URL from predecessors

• Basically move combiner's functionality into Data Store