Slow initial start with huge collection
eboye opened this issue · 55 comments
I've got a very big collection of music. When closing and starting the app it hangs for few minutes like this:
Is there a way it can load items from cache and sync in background?
This is the folder I'm using as collection in MusicPod
~45.000 files
hm yeah we could create an index or something
45k is something i didnt have in the beginning
I will think about this!
If it really a few minutes? Or seconds?
Nope, I'm not using snap. I'm using the appimage.
First startup took around 7 minutes and it showed a message I've got 29k songs and if I want to save the cache. But after that the GDM broke and I needed to reboot.
I'm trying to start it again and it's rebuilding indexes again. Here's the screen recording I've made until the crash.
Screencast.from.2023-11-17.00-47-08.webm
I'll try to save the cache again once it's built. The crash may be due to recent update on of kernel modules and without reboot (I hope).
I'll report back the results if the cache saving passes.
I'll try to run the flutter project to have some logging if the crash happens again.
It didn't crash this time:
And closing takes roughly 5 seconds.
Next start is around minute or so with blank window while loading. I'll upload the screen record. But it's considerably faster.
OK, now closing is almost instant. Here's the startup screen recording:
Screencast.from.2023-11-17.01-05-58.webm
Here's the app profile startup. The first item in dart debugger says it took 11s. I think you can import it in your profiler or something. It may help.
dart_devtools_2023-11-17_01 11 53.406.json
Thank you for the detailed report!
I'll move the index building and index loading onto a second CPU threat (called isolate in dart) to not block the UI
Will ping you again when I'm done
@eboye okay :) check main again, I think this is now good for both extremes: very big library and cache wanted and people with small library or people wo dont use the local audio feature at all (and only use radio and podcast feature)
Well, it's kinda worse then the last version.
It acts like it's rebuilding every time even though it's set to use caching.
Screencast.from.2023-11-18.01-25-04.webm
Damn. Okay I think I need to rethink this totally Oop approach to audio files and their metadata. I think the problem might be the images in the mp3s
okay @eboye I think now it is better :) I made a mistake
to reset the settings delete .config/MusicPod/ or snap/MusiPod
slow machine today so slow video
Screencast.from.2023-11-18.11-42-53.webm
In my ~2k audio collection this cache slows down the start
But I think for 25k it should improve the start-up now (I hope)
if this finall works better for your huge library then I will set the threshold to ask for the cache maybe to 5k
Great! Not ideal, but I think this is much more usable :)
Screencast.from.2023-11-18.12-31-39.webm
I think we can close this issue. I have some experience with flutter (but mobile only), so I'll see if I can find some tweaks here and there to speed up the process. If I do, I'll make PRs :)
Thanks!
Thanks! This was after loading the cache?
If yes then hmmmmm I really wonder how to improve this. Eventually I should not store the in memory images in the cache at all. I need to rework the medata thing then. I have an idea but this will take some time
No let's keep this open as this is clearly still not acceptable
I guess other programs load faster even with 25k files?
Yes, with cache loaded.
Well, native ones are much much faster (C, Rust, Python...), electron is mixed bag.
Strawberry and Clementine are extremely fast for example. But the UI is garbage IMHO 😀
OK, but I really like the UI on this one, so I'll stick with it for now.
@eboye hey again. I've made another mistake and accidently created the cache twice... sooooo it should in theory be 2x fast for you now :) if you have time would appreciate if you re-check main / snap from edge
Will do. I'm using an appimage that I build myself 😁
I'll report back
Initial start is instant.
First start never finished indexing or what.
I've deleted .config/musicpod/
It started with loading icon (not sure why)
I've picked the folder. It indexed in a minute or two (much faster then previously)
Note: Scroll is short (for something like 50 items, when end reached it adds up)
Saved index
Closing takes ~10 seconds
Next start is instant
Loading of local audio takes almost the same time as first indexing (minute or two).
Is it parsing the index that long or is it reindexing?
When it finished, closing was instant.
Initial start is instant.
First start never finished indexing or what.
I've deleted .config/musicpod/
It started with loading icon (not sure why)
I've picked the folder. It indexed in a minute or two (much faster then previously)
Note: Scroll is short (for something like 50 items, when end reached it adds up)
Saved index
Closing takes ~10 seconds
Next start is instant
Loading of local audio takes almost the same time as first indexing (minute or two).Is it parsing the index that long or is it reindexing?
When it finished, closing was instant.
Sounds good except that the loading of the local audio takes again same time. If I understand this correctly?
Yup, when the index is saved, loaded from cache, it takes some time, maybe not as indexing, but still it's slow to show something. I'll record the video if you need.
I expect it to be under 10-15 seconds.
Yup, when the index is saved, loaded from cache, it takes some time, maybe not as indexing, but still it's slow to show something. I'll record the video if you need.
I expect it to be under 10-15 seconds.
video would be nice
Really hard to test this because I dont have such a big collection. I could write a test for this maybe.
Make sure to delete the config folder of musicpod first
Hmm, I'm debugging now and it seems that ~/.config/musicpod is recreating when the app is started. But deleting it doesn't delete the index.
In ~/.cache/images are only images, there is no index files.
I look for MusicPod folder, but it doesn't seem to be in nor config nor cache. I'm not using snap, so there is no ~/.snap folder.
this seems to be different for app images then
in the snap it is one location
serch for settings.json or localAudioCache.json
Well, it is in ~/.config/musicpod
I'll try to investigate further and report back.
hey @eboye
I've cleaned up a lot of bad code. Especially today with localaudios could bring some improvements.
At least on a very weak machine I have here the local audio loading and searching is now much faster
Best regards 👋
Well, not sure if it's faster. This may be the issue:
the cache file is 2.3GB
In the file I see that you are caching images as base64 ... maybe you could exclude caching the images and load them lazy or something in the UI from the files itself?
Well, not sure if it's faster. This may be the issue:
the cache file is 2.3GB
In the file I see that you are caching images as base64 ... maybe you could exclude caching the images and load them lazy or something in the UI from the files itself?
Yeah.. it's sadly not possible to exclude the memory images while reading the metadata.
I think I need to work on this in the library I use for this
Well, not sure if it's faster. This may be the issue:
the cache file is 2.3GB
In the file I see that you are caching images as base64 ... maybe you could exclude caching the images and load them lazy or something in the UI from the files itself?Yeah.. it's sadly not possible to exclude the memory images while reading the metadata. I think I need to work on this in the library I use for this
I've made a PR that can improve this, but then it lacks images later on
hey @eboye
I want to try a different approach first
I switched away from metadata god package to the id3tag package
it promises to "Reading ID3 meta data without having to load the entire file into memory"
I have also added a test now which I would like to expand to your extreme test scenarios of 25k mp3 files.
Could you check out the main branch / snap edge and tell me if there is any improvements?
best would be if you would delete the settings files and your cache and not use the cache for the moment, I think the way I made the "cache" doesnt make much sense anyways
I'm not sure if this is the same bug, and don't want to create a duplicate if so, but for me, Musicpod simply shows a big, flashing, music icon:
It never moves past this point, and running it from the CLI doesn't give useful output, so I can't tell what's actually happening. I'd love to use it to replace Rhythmbox, but I can't even get into the interface. Any idea what could be causing this?
My library is pretty huge, btw, but I never even got to the point where I could configure the application, so I can't tell it what to load or not.
Update: So, I got frustrated with waiting, and decided to try renaming my music and video folders temporarily. This worked for getting into Musicpod, but now it's still taking forever to actually load anything. A more familiar workflow, as happens in many similar apps (though, frankly, not all do this - and I wish they would) - is to have the application load these folders asynchronously. In other words, it allows the user to view and play files while it's adding the remainder.
Now, I don't know if this is a greater technical challenge, but it would at least help with the long wait times.
EDIT: I'm using the latest version in the edge channel.
EDIT2: If you try this and then restore the correct folder names, it hangs again, so this is NOT a workaround.
I've got the same issue due to a large music library size so I added a little bit at a time to the music folder to see where it breaks and I get the slow strobing music note at startup. I found it breaks at about 31GB - roughly much more than that it just strobes/fails. When I removed enough files it opened right up.
Is there a plan to increase music library sizes greater than 30GB? For what it's worth, I'm hoping it can handle 50GB if that's doable because I really like this program.
Thanks in advance - really nice front end to MPV.
I've got the same issue due to a large music library size so I added a little bit at a time to the music folder to see where it breaks and I get the slow strobing music note at startup. I found it breaks at about 31GB - roughly much more than that it just strobes/fails. When I removed enough files it opened right up.
Is there a plan to increase music library sizes greater than 30GB? For what it's worth, I'm hoping it can handle 50GB if that's doable because I really like this program.
Thanks in advance - really nice front end to MPV.
interesting find!
yes I will completely rework the way local audio is being handled with a database to make it work for whatever size people have. but I can't give you an ETA on that. I will keep this issue updated
@eboye I think i finally found a solution for everyone 😆
I reworked as you recommended to not load all music with the images on start. Instead they are loaded on demand when they show inside the UI
However for people that have small to medium sized libs I would like to keep the load cache on app start feature (remade it now properly) and add a dialog to confirm that the cache wont be saved for big lib people
Give it a try y'all, I think I honestly fixed it fingers crossed 😆
1.5.2 is out
(for ubuntu snap users, its in the stable channel now)
Still stuck on loading at the start for me. I think the solution here would be to scan asynchronously from the UI? At this point, it never loads for me, ever. If the library could be scanned separately from the UI, I could at least see the interface, play what has already been scanned in, etc. This is how many music apps work, and the expected behaviour for most users (I'd assume).
Still stuck on loading at the start for me. I think the solution here would be to scan asynchronously from the UI? At this point, it never loads for me, ever. If the library could be scanned separately from the UI, I could at least see the interface, play what has already been scanned in, etc. This is how many music apps work, and the expected behaviour for most users (I'd assume).
it is actually in a separate thread
how big is your lib?
lets wait for some other people to report, I am not sure yet if you are facing a second, yet not less annoying, issue I didnt think about yet
Right now my music folder is 52GB (been collecting for years between different systems since the 90s or so, go figure).
Well, 5min on every start for me
Screencast from 2024-08-05 10-38-07.webm
I'm guessing it's because of some wma file I have that trowed some warnings in one of my testings as they do not have proper images or ID3 tags.
Well, 5min on every start for me Screencast from 2024-08-05 10-38-07.webm I'm guessing it's because of some wma file I have that trowed some warnings in one of my testings as they do not have proper images or ID3 tags.
I wish it was 5 minutes for me lol
It doesn't even give any useful info on the terminal.
Well, 5min on every start for me
Screencast from 2024-08-05 10-38-07.webm
I'm guessing it's because of some wma file I have that trowed some warnings in one of my testings as they do not have proper images or ID3 tags.
Yeah this is what I thought
I need to be better at excluding bad files
It's instant for me with 16 GB mp3
I left it running for a few hours this morning to see if it would ever give any output (while running through strace), but it's no different from when running normally. I really don't know what it is doing, but the interface never loads past the spinning animation.
@rolandixor yup, that was the case for me also, I just removed the config/cache files and it started behaving normaly.
I haven't had the same success. I removed the musicpod folder altogether, to allow it to start fresh. It still does the same thing.
I'll add more debugging breadcrumbs
That's the only thing for me to understand why for some people it's not working
Unless you send me your computers xD
@rolandixor @eboye if you have nerves for this would appreciate to try the snap from edge (or build from main if you are no snapper), made another performance trick
@rolandixor @eboye if you have nerves for this would appreciate to try the snap from edge (or build from main if you are no snapper), made another performance trick
I'll give it a shot. Will it allow the interface to load before it's finished scanning? 😅
@rolandixor @eboye if you have nerves for this would appreciate to try the snap from edge (or build from main if you are no snapper), made another performance trick
I'll give it a shot. Will it allow the interface to load before it's finished scanning? 😅
No but I can change this too. Then wait with your test and I give you a ping
@rolandixor ok have a try please, your suggestion is in (loading local audio only when local audio page is selected) (edge channel in ubuntu snap)
The fix definitely allows the interface to load, however local audio still results in spinning endlessly. Perhaps loading the list of audio files in chunks (is this possible?) would help here. I don't know (in terms of actual methodologies) how Rhythmbox does it, but it allows you to start listening to what the app has already scanned even while it's loading the rest of your local music.
Some players don't let you play anything, but at least show a progress bar, so even if it'll take hours to load, you know what is happening. At this point, unfortunately, Musicpod doesn't say what is happening, so there's no way to know if it's stuck, if it's just taking a while, or if it's already loaded the library and something else is wrong.
My suggestion would be to at least show a progress bar, if loading the database incrementally is too much work.
The fix definitely allows the interface to load, however local audio still results in spinning endlessly. Perhaps loading the list of audio files in chunks (is this possible?) would help here. I don't know (in terms of actual methodologies) how Rhythmbox does it, but it allows you to start listening to what the app has already scanned even while it's loading the rest of your local music.
Some players don't let you play anything, but at least show a progress bar, so even if it'll take hours to load, you know what is happening. At this point, unfortunately, Musicpod doesn't say what is happening, so there's no way to know if it's stuck, if it's just taking a while, or if it's already loaded the library and something else is wrong.
My suggestion would be to at least show a progress bar, if loading the database incrementally is too much work.
Endlessly sounds like there is some other issue going on for you.
Which audio format(s) do you use?
I've got the same issue due to a large music library size so I added a little bit at a time to the music folder to see where it breaks and I get the slow strobing music note at startup. I found it breaks at about 31GB - roughly much more than that it just strobes/fails. When I removed enough files it opened right up.
Is there a plan to increase music library sizes greater than 30GB? For what it's worth, I'm hoping it can handle 50GB if that's doable because I really like this program.
Thanks in advance - really nice front end to MPV.interesting find! yes I will completely rework the way local audio is being handled with a database to make it work for whatever size people have. but I can't give you an ETA on that. I will keep this issue updated
It works and I love it!! My 47GB music library loads and all playable. Thanks all!!!!
I've got the same issue due to a large music library size so I added a little bit at a time to the music folder to see where it breaks and I get the slow strobing music note at startup. I found it breaks at about 31GB - roughly much more than that it just strobes/fails. When I removed enough files it opened right up.
Is there a plan to increase music library sizes greater than 30GB? For what it's worth, I'm hoping it can handle 50GB if that's doable because I really like this program.
Thanks in advance - really nice front end to MPV.interesting find! yes I will completely rework the way local audio is being handled with a database to make it work for whatever size people have. but I can't give you an ETA on that. I will keep this issue updated
It works and I love it!! My 47GB music library loads and all playable. Thanks all!!!!
interesting! thanks :) then it was the thumbnails after all! Something to investigate :) I keep the issue open for feedback or until all people suffering return with a positive outcom
Oh my gosh it actually works now! Still loaded for a while but i just got back to my computer and it's working! Thank you for continuing to work on this! Excited to finally try this app out now!