banzaicloud/spark-metrics

Add complete example

C-h-e-r-r-y opened this issue · 1 comments

C-h-e-r-r-y commented

First of all thank you for greate job you did.
But it is very hard to use this example.
The ideal solution wiil be to create docker images with prometheus, spark etc. Where all property files are put into correct places, all enviroment variables are set...
A spark-submit command with all it's argument can be put into some sh file so to see how all of this works use just do the following:

  1. Start docker containers (may be via docker-compose up command)
  2. connects to spark container
  3. invokes this sh file
  4. wait a couple of minutes when dummy spark jobs is finished
  5. see metrics in promethus

For now every user have to do all of this from scratch and spend a lot time. Especially when deals with metrics first time.

f1yegor commented

it's not a complete example, but our dockerfile looks like

COPY ./<>/spark-prometheus-metrics.conf $SPARK_HOME/metrics.properties
COPY ./<>/jmx-collector.yaml $SPARK_HOME/
COPY ./<>/spark-metrics-assembly-2.4.jar $SPARK_HOME/jars/

and then setting
"--conf", "spark.metrics.conf=/opt/spark/metrics.properties"
via submit or sparkConf.set itself.