Options for transformations on metrics stream
Closed this issue · 3 comments
Reviewing options that work well for our use case to run transformations (aggregation, groupby, math functions) on metrics streamed at kafka.
Kafka streams, KTable probably present as the best fit to read, transform and write back the metrics data to kafka.
Kapacitor (https://www.influxdata.com/blog/announcing-kapacitor-an-open-source-streaming-and-batch-time-series-processor/) provides efficient transformations using InfluxQL but does not support directly reading from a kafka source yet.
Please let us know for open source solutions that we could review for use.
Note that John M made Kapacitor work for exactly this use case. He reads from Kafka and pushes to Kapacitor. This gives him the ability to run InfluxQL queries (which are like Graphite functions). Will talk to him to find out more.
Closing, as this would be external to AA.
(We are working on this internally.)