spotify/flyte-flink-plugin

Add support for default logging properties

regadas opened this issue · 1 comments

Add support for default logging properties

As I understood, how this should work is; users pass the custom logging properties via task context and we set these custom logging properties on FlinkClusterSpec of Kubernetes operator.

we need to do the following to get this working,

  • Update the config.go to add this new property.
  • extract these custom logging properties here and pass it to BuildFlinkClusterSpec
  • In the resource.go, set this to FlinkClusterSpec object.
  • Above step requires to tweak the FlinkClusterSpec in the flink operator to hold the value.

Before doing this I would like to know the rationale behind this change. At which point and/or where we will use these properties?

After an offline discussion, got to know that what we really need is to set LogConfig in FlinkClusterSpec. the LogConfig in Flyte context is different from this. Have to add a new field in flinkIdl to get user-given values and pass it to flink opertor.