AbsaOSS/pramen

Add support for column comments in config transformations

Closed this issue · 0 comments

Background

Currently, you can define simple transformations as part of configuration like this:

transformations = [
  { col = "col1", expr = "lower(col2)" }
],

Feature

Add support for [optional] column comments in config transformations.

Also adding a comment can be the only operation of a transformation.

Example

transformations = [
  { col = "col1", comment = "A comment for existing col1" }
  { col = "col2", expr = "lower(col1)", comment = "A comment for new col2" }
],

This is useful if the table is exposed in Data Lake, Glue, or some other metastore.