/storm-metrics-reporter

A Storm metrics reporter module.

Primary LanguageJavaApache License 2.0Apache-2.0

storm-metrics-reporter

A pretty generic Storm metrics reporter, currently used to adapt Storm metrics to Yammer metrics API and report them to Graphite, inspired by storm-metrics-statsd.

Usage

Build the jar:

git clone https://github.com/staslev/storm-metrics-reporter.git
cd storm-metrics-reporter
mvn package install

Add a pom dependency:

<dependency>
  <groupId>com.github.staslev</groupId>
  <artifactId>storm-metrics-reporter</artifactId>
  <version>1.0-SNAPSHOT</version>
</dependency>

Configure your topology:

  1. Specify MetricReporter as the metric consumer.
  2. YOUR_GRAPHITE_SERVER and YOUR_GRAPHITE_PORT values.
  3. Specify the metrics you want to report by providing an appropriate regular expression, in the example below we're reporting any metric that has the words "execute", "latency" or "capacity".
  4. Specify your StormMetricGauge implementation, in the example we're using the provided SimpleGraphiteStormMetricGauge class.
final Config config = new Config();
config.put(MetricReporter.METRICS_HOST, YOUR_GRAPHITE_SERVER);
config.put(MetricReporter.METRICS_PORT, YOUR_GRAPHITE_PORT);
config.registerMetricsConsumer(MetricReporter.class,
                               new MetricReporterConfig("(.*execute.*|.*latency.*|.*capacity.*)",
                                                        SimpleGraphiteStormMetricGauge.class.getCanonicalName()),
                               1);

Screenshots

Graphite metric hierarchy

Execute Latency Metrics

Disclaimers

  • This project was inspired by storm-metrics-statsd, thanks jt6211!
  • The screenshots above were generated using a custom StormMetricGauge, not the one included in the sources. It depicts a particular Graphite naming convention (the host for instance, is not seen in the metric hierarchy chart as it's located higher in the hierarchy). Metrics naming styles are subject to change in the other environemnt.
  • storm-metrics-reporter currently supports reporting metrics to Graphite only, but it should be pretty straight forward to extend it to support other metrics reporting mechanisms.