Fantom-foundation/go-opera

Unique metric streams are unbounded

thingtheory opened this issue · 0 comments

Describe the bug

opera creates multiple metric streams (in this case, unique metrics) per epoch. This means that a continuous running node will create, and report to consumers, an ever increasing number of streams (until it restarts).

Example (prometheus style)

# TYPE opera_chaindata_epoch-213180_disk_read gauge
opera_chaindata_epoch-213180_disk_read 7820353

# TYPE opera_chaindata_epoch-213180_disk_write gauge
opera_chaindata_epoch-213180_disk_write 35209142

# TYPE opera_chaindata_epoch_213180_disk_size gauge
opera_chaindata_epoch_213180_disk_size 3982687

To Reproduce

Run opera with --metrics to enable metrics.

Expected behavior

The number of streams reported by opera does not increase over time.

There's probably a few ways to solve this:

  1. Don't report per epoch metrics
  2. Put the per epoch metrics behind the --metrics.expensive or new flag
  3. Only report metrics for the current or last N epochs

Additional info

I'm not familiar with how influxdb works, but this behavior causes prometheus memory usage to grow unbounded due to the increasing number of active streams.