Pushing 100% memory utilization after update to 1.9.6

[nickmonad]

After updating to 1.9.6, I've noticed our memory usage is pushing 100% utilization, even with config limits in place. I did not notice this behavior on 1.9.5, where I tried to keep the memory usage to about 75-80%

Here's a graph showing one of our nodes running (with 8GB of RAM)

And the associated config,

# https://github.com/cculianu/Fulcrum/blob/master/doc/fulcrum-example-config.conf
datadir=/mnt/fulcrum
bitcoind=<redacted>:8332
rpcuser=<redacted>
rpcpassword=<redacted>
stats={STATS_BIND}:8080
db_mem=6000 # instance has a maximum of 8GB
db_max_open_files=100
tcp=0.0.0.0:50001
max_clients_per_ip=-1 # we are behind a load balancer that loses client IP information, so we need to make this unlimited
check_db=true
anon_logs=true
hostname=<redacted>
peering=false
announce=false

We have two load-balanced nodes and each only has an average of 20 clients at any given time, with a max of about 40 each, so I don't believe the high memory usage is related to increased client connections

[cculianu]

Hmm.

Bitcoind memory pool has been huge as of late. Still 8GM seems high. The updates really didn't touch much -- they mostly dealt with optimizing how the node loads txns from mempool from the daemon. Also added a few RPC calls but those calls are immaterial from a memusage point of view as they just leverage existing APIs in creative ways -- nothing architecturally significant changed, in other words.

I am tempted to say this may be a coincidence -- the update came out around the same time that BTC was experiencing huge congestion in the mempool -- that was the reason for the update since I got annoyed at how long mempool synch would take (esp. on first run) and I noticed ways I could make it faster.

Now, with db_mem = 6000 the database will store more of the utxo tables in memory. And with a huge mempool more access to the database will cause it to cache its tables in memory more. This alone could explain the higher memory usage.

So... I would def. lower db_mem. You don't want to push it so high on an 8GM system. I would make it 2000 at most personally. You only have 20 clients you don't need to devote so much memory to caching db. Esp. if you are using a fast storage system like SD card or SSD.

Also -- are you running the release binary or building it yourself? In other words -- is your node using the linked-in jemalloc allocator or not?

[cculianu]

Just for reference this is the BTC memoool right now, as I type this. Over 140k txns and fees super high.

[cculianu]

Note if you lower db_mem and it still fails to be under control, and if you are running the release binary that uses the more efficient jemalloc -- you can disable the aggressive jemalloc "thread cache" for allocations -- by setting this environment variable before running fulcrum: MALLOC_CONF=tcache:false

The tcache:false option usually does reduce memory usage (at the expense of some CPU load).

[nickmonad]

@cculianu Thanks for the input. We are using the release binary directly from GitHub.

I'll start by lowering db_mem and monitor from there

[nickmonad]

@cculianu Lowering the db_mem to 2000 definitely helped. Although on an 8GB machine, with db_mem=2000, we are still hovering around 45% memory utilization (instead of the expected 25%) Not an issue for stability by any means, and the OS is contributing some amount here, but it does seem like Fulcrum is allowing allocations above the requested limit?

In any case, our immediate issue is resolved, and this can be closed!