Unique metric streams are unbounded
thingtheory opened this issue · 0 comments
thingtheory commented
Describe the bug
opera
creates multiple metric streams (in this case, unique metrics) per epoch. This means that a continuous running node will create, and report to consumers, an ever increasing number of streams (until it restarts).
Example (prometheus style)
# TYPE opera_chaindata_epoch-213180_disk_read gauge
opera_chaindata_epoch-213180_disk_read 7820353
# TYPE opera_chaindata_epoch-213180_disk_write gauge
opera_chaindata_epoch-213180_disk_write 35209142
# TYPE opera_chaindata_epoch_213180_disk_size gauge
opera_chaindata_epoch_213180_disk_size 3982687
To Reproduce
Run opera
with --metrics
to enable metrics.
Expected behavior
The number of streams reported by opera
does not increase over time.
There's probably a few ways to solve this:
- Don't report per epoch metrics
- Put the per epoch metrics behind the
--metrics.expensive
or new flag - Only report metrics for the current or last N epochs
Additional info
I'm not familiar with how influxdb works, but this behavior causes prometheus memory usage to grow unbounded due to the increasing number of active streams.