Buffer size too small
Closed this issue · 7 comments
"Two million bytes should be enough." You wish ;) This might work with truncated beatmaps (see ppy/osu-api#131), but look at this beast:
Krewella - Live for the Night (Nightstep Mix)
That's right. >2MiB beatmaps and they're ranked, too.
huh, what the hell how does such a short map have such a big .osu file ?_?
well I'll bump it to 8mb I guess
ohh storyboards can be embedded in the .osu file apparently, I thought they were always separated into .osb files
how about not reading the whole file into memory? xD
yes, this is just a temporary fix. if the files get more unpredictabily big I will read line by line. the C standard library buffers read syscalls anyways to something like 1 page of memory, so it won't be much slower.
another great option would be memory mapping the file, which is what I do for example in stubgen
https://github.com/Francesco149/stubgen/blob/master/linux.c#L42
https://github.com/Francesco149/stubgen/blob/master/windows.c#L65
might be even faster than reading the file like I do in oppai
To be honest, since this thing is running in Tillerinobot, I worry more about the memory usage than anything else. You'll eventually get through CPU shortage most of the time, but running out of memory will almost always make things crash.
well, linux doesn't actually allocate the pages until you touch them, and overcommit is allowed by default, so a bigger buffer shouldn't cause memory shortage unless there's actually that much raw beatmap data in memory.
but yeah, since storyboards can be embedded into .osu files (didn't know) I will prob end up reading it line by line.
memory mapping can save memory (opening the same file uses copy on write), but tillerino probably still has a massive amount of unique maps in memory
also remember that if you enabled the cache, it should only parse (and therefore put into memory) brand new maps