Extremely high memory usage when running an incremental scan on 200k+ files

Question

Closed this issue 5 months ago · 5 comments

gonic version: from source, 88e58c0

if from docker, docker tag:
if from source, git tag/branch: master

When running an incremental scan over 200000 files, memory usage ramps up to 5 gb or more.

I am running Arch Linux on a Raspberry Pi 4, using an sqlite database which is about 60 MB in size when the scan starts.

My music directory has 6 symlinks, which all point to various categories/genres.

The scan goes well until memory usage hits about 3-3.5 GB, then it starts increasing by about 100 MB a second.

At this point the scanned directory entries as per the log are printed significantly slower.

Answer 1 · 2024-03-08T21:23:15.000Z

woah that's crazy. was this always the case? if not, maybe you could try git bisect to find the commit which introduced this?

Answer 2 · 2024-03-08T21:29:12.000Z

No, it looks like this is fairly recent, I'd say the last 2-3 months.

Unfortunately I don't have much time these days, but I'll do a bisect when I can get to it, thanks for the tip. It might take me a while though.

Answer 3 · 2024-03-09T03:40:56.000Z

thanks! also I just pushed a commit which may help

Answer 4 · 2024-03-09T10:39:50.000Z

Huge thanks for this.

With 7cd1bee, memory usage stays around 100 MB during a scan and everything works as expected.

Answer 5 · 2024-03-09T19:33:15.000Z

thanks for reporting the issue 👌