opentraffic/architecture

create time binning for statsitics

Opened this issue · 1 comments

kpwebb commented

Can we develop a stats bucket that grows the time window to capture data in a reasonable way?

kpwebb commented

@bmander and I just agreed on the following approach:

  1. Traffic Engine emits a stream of stats observations for a segment with the following:
    segment identifier,utc time,speed

Every N minutes we rotate the traffic-engine stats stream store

  1. We build a stats aggregator that collects thr rotated stores every N minutes and aggregates into defined time bins.

Once a bin meets the criteria for being sharable (enough stats to anonymize) we ship the data and reset the bin. Bins without enough data are kept and their time window is expanded by N minutes when the next round of rotated stats are added.

Some bins will be "timeless" in that there aren't frequent enough samples to keep stats on time windows. In this case all samples just live in a grouped collection.