New mod portal changes broke things
Closed this issue · 15 comments
The mod portal has just recieved a massive overhaul, API included as they have gone away from basing the core of the site on AJAX calls.
Now, these api changes have also broken most tools relying on them, including this project 😢
Among the things I have noticed are broken:
-
Searching doesn't seem to care about the search term
-
Downloading results in a file named by some weird hash, rather than the proper filename. It is still downloaded correctly and working after renaming the file though.
Probably more, but those are the ones I have caught this far. Also, you still haven't pushed to npm after resolving #2 😄
Oh sorry, I totally forgot about that.
I might try to update this whenever I've got time. It will be tricky, because it's not documented for as far as I know. I'll have to rewrite a lot of things probably. I do want to maintain this project.
And may I ask where you're using this for? I'm just curious :)
Found some info regarding the searching issue!
< mickael9> looks like you can't use /api/mods?q=SEARCH to search mods now :(
I am using this for managing mods from the commandline, also downloading mods before joining a server so I don't have to mess with multiple installs. I really don't like using the browser for file management...
Oh yeah I get it, makes sense to use it that way. Managing mods in-game isn't really the greatest option either.
I'm basically just going to try to figure out which requests are being made while I'm using the mod portal website. That's how I figured it out in the past. I might also improve my unit testing and make sure everything works again.
I believe /u/bilka2 has posted some information about the portal API on github, he might have more info in case you want to ask him.
I'm not sure where to find that information, but I might ask him if I need help.
Found some more info from people talking about the changes maybe.
Yeah, it might be better to wait until they (finally) make a public API. That would make things easier anyway. The previous API wasn't public, but the community documented it on the wiki. I don't want to waste my time figuring it out right now, as it will probably change anyway.
Hey Daniel
I've got some good news and some bad news. The good news is:
The bad news is, searching mods is no longer possible. Apparently, the Factorio devs never used this functionality, because Factorio just gets a list of all the mods and searches in that list locally. That's probably why they removed it. Anyway, I might do something similar, so you can search mods again.
Fixing the problem with the file names was very easy. To fix the dependency problem, it now uses the local zip file from a mod to get it's dependencies. Reading mods from a save file, should work with 0.16 saves, but not with older versions.
The update is already published to npm and I'll try to find a solution for searching mods.
Okay, you can ignore the bad news. I fixed the searching problem by just requesting for the first 100000 mods and filtering them manually. I sorted them from most to least downloads. The parameter of the searchMods function is now just a query and not an object.
Great work, this made my day so much better!
Just wondering, is the speed of searches significantly impaired by having to fetch all of the mods? How many mods are there even?
It used to be very slow and I expected it to still be very slow, but it has improved a lot. You may have noticed that when you try to download mods in-game, it loads a lot faster now too. I don't know how many mods there are, but I'm pretty sure it's less than 100000. I might have to change that number in a couple of ages xD. You can try this link if you want to see how fast it is: https://mods.factorio.com/api/mods?page_size=100000
According to my dev console, there are 2535 mods, which results in 1.1 mb of data which downloads in 3 seconds. For a full 100k mods, that would be about 40mb of data, which is still not too bad for something that gets downloaded so rarely. And this is probably a thing we could cache pretty safely.
Yeah, I could make it so that it only downloads once when you authenticate and caches it so it doesn't have to download it again for each search query. But 1.1 MB isn't that bad I suppose.
Personally, I think that caching should be done by the user. You might want to for example recheck to see if a mod has been updated etc, and having to deauthenticate for that would be confusing.
Yeah, I agree. I could make a function to reload cached data. The CheckUpdates function is not affected by the changes of the mod portal, because you can still find information about a specific mod if you have the exact name. Every time you call CheckUpdate it will go to a link like this one: https://mods.factorio.com/api/mods/boblogistics . It doesn't load all mods to check updates. The fix we're talking about was only for search queries. I think the chance a new mod gets posted right after you authenticate is very small. If download a mod, it will always try to download the latest mod, it doesn't matter if you just reloaded the list of all the mods. It would still make sense to have a function to update the cached data anyway. I hope any of what I said makes sense.