Server RAM usage increasing

Question

Server RAM usage increasing

Opened this issue 9 months ago · 10 comments

HEllo,
After 5 days of server start the ircd process is consuming 1.8 GB of RAM.
High peak was 5 GB after 30 days. What can be the issue?
Running Debian 12 64 bit and Nefarious2 git version.
Thank you.

Answer 1 · 2024-02-24T22:18:51.000Z

I have observed this too on a similar set up. I'm trying to isolate what may be triggering this so I wrote a node script that will trigger periodic core dumps once ram usage or rate of increase hits a certain threshold to see if we can figure what is possibly being leaked. Right now I'm awaiting for the mem runaway to trigger, and I'm afraid that'll be a waiting game since I don't know what might be triggering it. Is it always reproducible for you? Are you on a network or a single server? Have you noticed when it seems to trigger; perhaps after a user connects and sets a certain user mode? When the runaway happens, I'll notice the leak is pretty rapid, 10 percentage points of increase in an hour so until I hit the machine cap

…

On Sat, Feb 24, 2024, 16:28 ZioNeL ***@***.***> wrote: HEllo, After 5 days of server stats the ircd process is consuming 1.8 GB of RAM. High peak was 5 GB after 30 days. What can be the issue? Running Debian 12 64 bit and Nefarious2 git version. Thank you. — Reply to this email directly, view it on GitHub <#76>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA252YMJN5IUDC3LZ2MRX5LYVJLQDAVCNFSM6AAAAABDYJOOC6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGE2TENBYHEZDQMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

Answer 2 · 2024-02-24T22:49:14.000Z

We have a small network, 2 main servers and also 3 servers for services.
Not so much of activity. What i can tell is that with each day that passes the RAM consumed is more and more high.

Answer 3 · 2024-02-24T23:09:13.000Z

Out of curiosity (and in an attempt to rule something out), what distro are you using? and what openssl library version do you have installed?

Answer 4 · 2024-02-24T23:18:33.000Z

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm

OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023
3.0.11-1~deb12u2

Answer 5 · 2024-02-24T23:25:51.000Z

Thanks, that ruled out the wild goose I was chasing.

Answer 6 · 2024-02-26T22:56:48.000Z

Does the increase in memory use happen immediately after you restart, or does it run for a while with very little, and then start to get bigger after some triggering event?

Answer 7 · 2024-02-27T19:49:32.000Z

is growing little by little, for example 3 days ago RAM usage was 1.84 GB, now is 2.74GB

Answer 8 · 2024-04-01T14:46:32.000Z

I`ve noticed that RAM increase is only on running servers where services are linked. Due to lots of net splits. Everytime a netsplit is taking place also ram increase is happening.

Answer 9 · 2024-05-28T20:49:10.000Z

@ZioNeL Just to update you, I'm still waiting for my server to go into a mem-leak state. I have a system that monitors the RAM usage, and if it exceeds a certain threshold or increases too quickly, it generates memory dumps.

Sadly, since we don't know what triggers it, it's a waiting game.

Answer 10 · 2024-11-26T21:09:17.000Z

Hey there, @ZioNeL -- any more reoccurrences of this bug? If possible can you get a memory core dump of the ircd process when it happens, and then get a strings output of it? I'm investigating different things and have a couple ideas and theories.