metricq/metricq-sink-nsca

Improve handling of non-monotonic metric values

Closed this issue · 1 comments

In theory, all metrics should only produce strictly monotonically increasing timestamps.

Practice is different though. Currently, the checker fails with:

ValueError: Failed to update state history of 'elab.ariel.s1.package.power.1Hz'
[2021-02-03 15:08:57,053] [ERROR] [metricq_sink_nsca.check] Unhandled exception when checking values for 'elab.ariel.s1.dram.power.1Hz'
Traceback (most recent call last):
  File "/home/service/envs/nsca/lib/python3.9/site-packages/metricq_sink_nsca/state_cache.py", line 417, in update_state
    metric_history.insert(time=timestamp, state=state)
  File "/home/service/envs/nsca/lib/python3.9/site-packages/metricq_sink_nsca/state_cache.py", line 133, in insert
    raise ValueError(

and cannot update the state of the corresponding metric.

Options:

  1. Ignore non-monotonics
  2. Ignore them, but make the behaviour configurable
  3. Keep it this way.

We decided to settle for option 1 (for now). Not ignoring non-monotonics is hard because of false positives/negatives (cf. #16).