serenity-rs/songbird

Possible memory leak in Serenity 0.12 and Songbird 0.4.

abokado0107 opened this issue · 4 comments

Songbird version: (version)
0.4.0-rc

Rust version (rustc -V): (version)
1.74.0

Serenity/Twilight version: (version)
Serenity 0.12.0-rc

Description:
When a large number of small audio files are played back, CPU utilization increases and memory usage explodes after a certain period of time.
image

Steps to reproduce:
Rust installation is omitted.

$ sudo apt update && sudo apt install libopus-dev build-essential autoconf automake libtool m4 pkg-config libssl-dev -y
$ git clone https://github.com/abokado0107/tts-memory-leak
$ cd tts-memory-leak
$ cargo build --release
  1. Build the verification repository.
  2. Enter token in .env and run it.
  3. Type !join on Discord to invite the BOT to the voice channel.
  4. After a few days, the memory usage will explode and crash in a short time.
    It crashes faster if you have it connected to several VCs at the same time.

Verification environment
CPU: ARM Ampere
CPU: 2 core
RAM: 4 GB
OS: Ubuntu 20.04
IaaS: Oracle Cloud Infrastructure
Repositories for verification: https://github.com/abokado0107/tts-memory-leak

I've been running my own bot continuously for a day or so which also plays back many small audio files; I'm not seeing my CPU or memory substantially increase over time in a single server. I'll keep you posted if anything changes on that front.

I don't know if this is directly related to this issue, but after several days of running my application, I am accumulating tasks that never finish.
This image is displayed by tokio-console.
image

I am not familiar with the structure of serenity or songbird, but it seems that tasks generated by songbird's task.rs and serenity's tokio.rs remain relatively well.

I need a little more context; one of each driver/scheduler/task.rs:112 is expected for every Call without live audio playing, If there's a big difference between those two quantities then that would indicate a task leak. If you're joining/creating Calls but not removing, this would be expected, otherwise I'm very interested.

Thanks for letting me know.
I checked in the production environment and Call was able to remove properly.
I output the following counts to the debug log and the call was increasing and decreasing properly.

if let Some(manager) = songbird::get(ctx).await {
    let call_count = manager.iter().count();
}

image

As for the scale of the application, the number of simultaneous VC connections at 30k guilds is about 2k at peak time.