bifrost-io/bifrost

Memory leak

Closed this issue · 4 comments

stakeworks commented

Ubuntu: 20.04 LTS
8 cores
30 GB RAM
bifrost 0.9.44-7af39eb

After upgrade to bifrost 0.9.44-7af39eb memory usage increases a lot! (the memory issue also happened after upgrade to robonomics 2.0.0-17b47a4bd7c)

image

image

Don't know if it's related but also getting these messages:

Labels
alertname = UnboundedChannelPersistentlyLarge
chain = bifrost
entity = mpsc_finality_notification_stream
instance = localhost:19635
job = bifrost-kusama-cmn12
monitor = CMN12
severity = warning
Annotations
message = Channel mpsc_finality_notification_stream on node localhost:19635 contains more than 200 items for more than 5 minutes. Node might be frozen.

Akagi201 commented

It seems that it is releated to this issue paritytech/substrate#11604

koute commented

If you have the following at the start of your logs:

[Relaychain] BEEFY is still experimental, usage on a production network is discouraged.

...and you're not actually using BEEFY then it's most likely the same issue.

If so here's a fix you can test out: paritytech/substrate#11694

stakeworks commented

Looks like it's been solved in version 0.9.52, memory usage is still normal after an hour. Will keep an eye on it for the next days.

image