AbsaOSS/pramen

Pass operation definition config to the transformer constructor instead of the full config

Closed this issue · 0 comments

Background

Currently, you can define a transformer like this:

class MyTransformer(conf: Config) extends Transformer {
}

In this case the full pipeline configuration is going to be passed to the transformer. This is not convenient when you want to parse config of the transformer itself, since the transformer config is inside the array of operations:

pramen.operations = [
  {
    name = "Transformation1"
    type = "transformation"

    class = "com.example.MyTransformer"

    my.configuration.key1 = "abc"
  },
  ...
]

Feature

Pass operation definition config to the transformer constructor instead of the full config.

Proposed Solution

in order to maintain compatibility with transformer definitions that rely on the previous behavior, allow getting full configuration as well.

All following 3 classes should be acceptable:

class MyTransformer1 extends Transformer
class MyTransformer2(transformerConfig: Config) extends Transformer
class MyTransformer3(transformerConfig: Config, fullConfig: Config) extends Transformer