banzaicloud/spark-metrics

spark metrics namespace configuration issue

tkrasnoperov opened this issue · 2 comments

According to the documentation at https://spark.apache.org/docs/latest/monitoring.html, under Metrics, we can set the spark.metrics.namespace configuration to something like ${spark.app.name}, which presumably is expanded in the the metrics sink to be the value of that variable. We opened up our spark-defaults.conf file and added the following line.

spark.metrics.namespace					     ${spark.app.name}

As a result, our metrics changed from

app_20180802190042_0119_driver_DAGScheduler_stage_runningStages{instance="",job="app-20180802190042-0119",role="driver"} 1

... to this ....

Parse_AUT1_records_driver_DAGScheduler_stage_runningStages{instance="",job="${spark.app.name}",role="driver"} 1

The actual name of the metric changed successfully (Parse_AUT1_records is in fact the name we expected), but it seems the job tag did not expand properly (says ${spark.app.name}). We looked into the source code and are pretty sure there is an issue here in https://github.com/banzaicloud/spark-metrics/blob/master/src/main/scala/com/banzaicloud/spark/metrics/sink/PrometheusSink.scala#L77 on line 77. The name seems to be taken verbatim as a string and not expanded as a variable.

@tkrasnoperov thanks for catching and reporting this. We'll look into this issue.

@tkrasnoperov please give a try using version spark-metrics_2.11-2.3-1.0.1.jar