AbsaOSS/pramen

Make Spark Application name configurable

Closed this issue · 0 comments

Background

Currently, Spark Application name of the Pramen job is set either:

  • by Pramen is Spark session does not exist
  • by a calling app if Spark session is created by a different app
  • only from command line when using Yarn in cluster mode

Spark Application name looks like this:

Pramen - ${pimeline.name}

Feature

Allow customizing Spark Application name by making it configurable

Example

pramen.spark.app.name = "My app - ${pramen.pipeline.name} on ${praman.environment.name}