Possible memory leak in Serenity 0.12 and Songbird 0.4.
abokado0107 opened this issue · 4 comments
Songbird version: (version)
0.4.0-rc
Rust version (rustc -V
): (version)
1.74.0
Serenity/Twilight version: (version)
Serenity 0.12.0-rc
Description:
When a large number of small audio files are played back, CPU utilization increases and memory usage explodes after a certain period of time.
Steps to reproduce:
Rust installation is omitted.
$ sudo apt update && sudo apt install libopus-dev build-essential autoconf automake libtool m4 pkg-config libssl-dev -y
$ git clone https://github.com/abokado0107/tts-memory-leak
$ cd tts-memory-leak
$ cargo build --release
- Build the verification repository.
- Enter token in .env and run it.
- Type
!join
on Discord to invite the BOT to the voice channel. - After a few days, the memory usage will explode and crash in a short time.
It crashes faster if you have it connected to several VCs at the same time.
Verification environment
CPU: ARM Ampere
CPU: 2 core
RAM: 4 GB
OS: Ubuntu 20.04
IaaS: Oracle Cloud Infrastructure
Repositories for verification: https://github.com/abokado0107/tts-memory-leak
I've been running my own bot continuously for a day or so which also plays back many small audio files; I'm not seeing my CPU or memory substantially increase over time in a single server. I'll keep you posted if anything changes on that front.
I don't know if this is directly related to this issue, but after several days of running my application, I am accumulating tasks that never finish.
This image is displayed by tokio-console.
I am not familiar with the structure of serenity or songbird, but it seems that tasks generated by songbird's task.rs and serenity's tokio.rs remain relatively well.
I need a little more context; one of each driver/scheduler/task.rs:112
is expected for every Call
without live audio playing, If there's a big difference between those two quantities then that would indicate a task leak. If you're joining/creating Call
s but not removing, this would be expected, otherwise I'm very interested.
Thanks for letting me know.
I checked in the production environment and Call was able to remove
properly.
I output the following counts to the debug log and the call was increasing and decreasing properly.
if let Some(manager) = songbird::get(ctx).await {
let call_count = manager.iter().count();
}
As for the scale of the application, the number of simultaneous VC connections at 30k guilds is about 2k at peak time.