[FR] Cache ScoreSaber Results Locally
Opened this issue · 6 comments
Every time I open the PP tool, I have to wait a few minutes for 100+ pages of results to load. Since the ScoreSaber API can return the newest scores first, it would be great if the app could be configured to cache old scores locally and only load the new ones (i.e., stop querying ScoreSaber when a score that has already been seen is returned and load the remaining scores from local storage).
Thanks for the suggestion ! So the newest scores endpoint is already used for the "refresh" button in the top left after loading your scores. My only worry with using it for everything is that things like the weighing on some songs could change and that would cause issues. I will play around though and see if I can come up with something that avoids any potential issues.
Ok it should now store the scores for up to 5 users (the ones in your history).
Since I'm still quite scared about caching data, it for now only uses that cache if there has been no changes to the ranked maps file. Unfortunately the ranked maps file also include things like upvotes and downvotes of each map, which means it will likely be changed at least once a day when we update those,
If everything goes well I'll probably consider loosening the use of the cache. My main real issue is if someone played a map in the past and then that map gets ranked, the map would not appear at the top of their recently played list and it wouldn't be in their cache (we only get the scores with pp to make things faster). But I could maybe keep two different update timestamps, adding one that would ignore insignificant changes like the number of plays or the number of upvotes, which means I would only have to refetch the entire song list only when maps get actually ranked.
I'm sure you've considered this but could you just store all player scores (ranked & unranked) locally? If you're fetching scores by "recent" you will have to fetch both ranked and unranked scores anyways, so you wouldn't need to reload the scores even if a song was newly ranked.
Yeah my issue with that is most people will have a few hundreds ranked plays, which can already take a while, but the total number of plays often goes up to multiple thousands (Tseska has over 6k plays currently) and I would like to avoid having the site take over 3 times as long to load even if it's only the first time. But it might be worth it in the end so I'll keep the option in mind.
The other issue that I just realized is that the rank on the song is going to become inaccurate. Since this comes from the scoresaber side, caching the request means the value won't be updated. But it could change when someone else beats the score, without the player themselves updating their score. This is an issue in some cases for the "score at rank" sorting, because if the site thinks that you are already at the desired rank or above, it will not do an extra scoresaber request to check what the goal score is. And I don't think there's any way to have the updated value except doing the regular full load
Does ScoreSaber allow you to only pull the ranked plays? My impression was that you would have to load all of the plays (ranked & unranked) anyways when you load by "new" in the ScoreSaber API.
So I do not use the "new" API at the moment as it is not officially out and could change / break at any moment without warning, so Umbra has recommended against it. Currently the front-end parses the scoresaber html pages instead, but that is roughly equivalent.
And yes indeed when sorting by new you cannot get only ranked maps, which is part of the problem. Currently, for the initial load, peepee gets the list of scores ordered by "top plays", which means we can stop once we reach a score with no pp (meaning the map is unranked, and all the maps below it are too), and avoid having to make too many calls. Recent scores only becomes the more interesting sorting method after that because we don't have to go through the whole list.
However, even if there was a way to get only ranked play, that wouldn't solve the bigger issue of not refreshing the ranks unless we get the whole list again (which would make it irrelevant whether we use the recent or top sort).