Filtering spark metrics before writing in prometheus

[gsumar]

Hello,

I am trying to create my custom Prometheus sink, in order that I can filter some of the metrics that are exposed to Prometheus.

To do that I am writing my own PrometheusSink that has a Reporter class in it, and extends the ScheduledReporter class.

class PrometheusSink(
                                 val property: Properties,
                                 val registry: MetricRegistry,
                                 securityMgr: org.apache.spark.SecurityManager)
  extends Sink with Logging {

  protected class Reporter(registry: MetricRegistry)
    extends ScheduledReporter(
      registry,
      "prometheus-reporter",
      new CustomMetricFilter(),
      TimeUnit.SECONDS,
      TimeUnit.MILLISECONDS) {

That filter is used to generate the report:

  public void report() {
        synchronized (this) {
            report(registry.getGauges(filter),
                    registry.getCounters(filter),
                    registry.getHistograms(filter),
                    registry.getMeters(filter),
                    registry.getTimers(filter));
        }
    }

I changed the filter in the way that it return the values that I need to depending if the metric need to be filtered or not.

public class CustomMetricFilter implements MetricFilter {
    /**
     * Matches all metrics, regardless of type or name.
     */
    public boolean matches(String name, Metric metric) {
        return false;
    }
}

The result that I was expecting is that all the metrics will be filter in my application, and none of them will appear in prometheus. But surprisingly, no matter what I write in the matches function of the class. Always is returning all of them.

It seems that the filter is not taken into account and it returns the implementation by default.

public SortedMap<String, Counter> getCounters() {
        return getCounters(MetricFilter.ALL);
    }

[gsumar]

I just tested my filter like this:

      println(registry.getCounters.keySet().size())
      println(registry.getGauges.keySet().size())

      println(registry.getCounters(new DataLakeEtlMetricFilter()).keySet().size())
      println(registry.getGauges(new DataLakeEtlMetricFilter()).keySet().size())

And is working OK. So it seems that is not taking the one that is given as an argument in ScheduledReporter.

[gsumar]

It seems that in DropwizardExport is not using any filter,

@Override
    public List<MetricFamilySamples> collect() {
        ArrayList<MetricFamilySamples> mfSamples = new ArrayList<MetricFamilySamples>();
        for (SortedMap.Entry<String, Gauge> entry : registry.getGauges().entrySet()) {
            mfSamples.addAll(fromGauge(entry.getKey(), entry.getValue()));
        }
        for (SortedMap.Entry<String, Counter> entry : registry.getCounters().entrySet()) {
            mfSamples.addAll(fromCounter(entry.getKey(), entry.getValue()));
        }
        for (SortedMap.Entry<String, Histogram> entry : registry.getHistograms().entrySet()) {
            mfSamples.addAll(fromHistogram(entry.getKey(), entry.getValue()));
        }
        for (SortedMap.Entry<String, Timer> entry : registry.getTimers().entrySet()) {
            mfSamples.addAll(fromTimer(entry.getKey(), entry.getValue()));
        }
        for (SortedMap.Entry<String, Meter> entry : registry.getMeters().entrySet()) {
            mfSamples.addAll(fromMeter(entry.getKey(), entry.getValue()));
        }
        return mfSamples;
    }

[stoader]

Hi @gsumar , you are right the DropwizardExports.collect() is not taking into account any metrics filters. A solution that I can imagine is to derive from DropwizardExports and override the public List<MetricFamilySamples> collect() method such as to take into account a metrics filter which can be passed in via constructor parameter of the derived class.

What do you think?