AbsaOSS/pramen

Allow the ability for multiple transformers to write to the same table

Closed this issue · 0 comments

Background

This is useful, when the transformation logic differs, say, on the day of week.

Feature

Allow the ability for multiple transformers to write to the same table.

Proposed Solution

  • The dependency resolution logic should mark a table as available only after all transformers that write to it are finished and successful.
  • Validation should be added to ensure multiple transformers do not write to the same partition.