Improve handling of non-monotonic metric values
Closed this issue · 1 comments
phijor commented
In theory, all metrics should only produce strictly monotonically increasing timestamps.
Practice is different though. Currently, the checker fails with:
ValueError: Failed to update state history of 'elab.ariel.s1.package.power.1Hz'
[2021-02-03 15:08:57,053] [ERROR] [metricq_sink_nsca.check] Unhandled exception when checking values for 'elab.ariel.s1.dram.power.1Hz'
Traceback (most recent call last):
File "/home/service/envs/nsca/lib/python3.9/site-packages/metricq_sink_nsca/state_cache.py", line 417, in update_state
metric_history.insert(time=timestamp, state=state)
File "/home/service/envs/nsca/lib/python3.9/site-packages/metricq_sink_nsca/state_cache.py", line 133, in insert
raise ValueError(
and cannot update the state of the corresponding metric.
Options:
- Ignore non-monotonics
- Ignore them, but make the behaviour configurable
- Keep it this way.