Tracking Rust rewrite (feedbacks are welcome)
mindstorm38 opened this issue · 45 comments
I'm currently quite bored with Python packaging and environment in general. This is just an idea and I know the work is really huge (already tried PoC in the past), so this is a really long-term discussion. Here is a little summary of pros/cons, and the prerequisites we want. Feel free to give feedback.
Prerequisites
- Exactly the same CLI, no breaking changes.
- Support Windows x86, x86_64, Aarch64 (don't know if there is a desktop on this arch), macOS x86_64, Aarch64, Linux/BSD x86, x86_64, armv7/8, Aarch64.
- Keep this repository.
Pros
- Bye bye Python (including dependency hell).
- No dependency hell in Rust (so better CLI, this whole idea originates from me trying to switch to click).
- Transparent change for users.
- Possibility of Python binding, you can keep updating CLI/API with
pip. - Lower-level native API (with .so, and for Rust dev, obviously, bringing PMC to a wider audience).
- Faster (this was not a big deal with PMC, but it's good to take).
- Standalone binary (potentially dynamically linked to its sibling .so).
- Better inclusion in the UNIX standard directory structure, potential man-page (even if I prefer exhaustive help messages, support for GNU language?).
- Available to many Linux distros (was already the case with pip, anyway...)
Cons
- I'm an annoying rustacean.
- Less portability, and I know that it has been really important in PMC adoption. In my opinion, this will not be that hard to have a great portability, we already have knowledge of important OS/Arch to support,
pipand other package managers can help us to make this transparent to users. - Rewriting is really likely to bring old bugs back, however we already have a good test suite for this.
- Takes long time, but we already have the knowledge.
I really believe this idea is worth pursuing.
While your argument is that a Rust rewrite would be less portable than current Python implementation (since you can run Python almost anywhere), I don't necessarily think this will be a problem.
If anything, a proper Rust rewrite should be more portable in the sense you could provide fully static binaries with no external dependencies - this would essentially eliminate what I think is the only limitation of PMC as it stands now - the need for a working Python installation. With a fully static binary, you would not need to depend on anything at runtime.
The external Python dependency is of course not a problem on Linux (given most distros have python preinstalled etc), but I assume it might be a real problem for the average Windows user.
The fact that PMC is implemented with the UNIX philosophy of "do one thing and it do it well" (which is why its so good, tbh), it would make sense for the 'core' of the launcher to be available as a Rust crate/shared lib, and the CLI interface as a thin wrapper of that. This would also allow users to write their own GUI wrappers etc.
tl;dr: The lack of dependency hell + ability to do static binary builds alone make this effort worthwhile, imo.
Thank you for this valuable opinion 👍 Your point on portability is right, and I never thought of Windows, but this is really true for non-python-dev users. And yes, the more I work with UNIX systems (Linux), the more I want the launcher to precisely follow UNIX philosophy. And since it is now on AUR, this is more relevant than ever because I really don't like package manager over python, because of the interaction between a stateful (any distro manager) manager and a stateless one (pip), this can lead to really annoying situations if you forget that a package was not installed via pip.
If this rewrite happens, it would be great if CLI args could be entered in any order. I haven't any experience with rust, but I know that there are great CLI arg parsing libraries on crates.io.
Yes that would definitely be the case, for now I'll be going on with the well-known "clap" crate. It is quite heavy but also really powerful. Arguments will be accepted in any order within the sub-command they are defined in, exactly like the current behavior with the python implementation. I've seen some projects interfacing with portablemc using the command line, so I really don't want it to break, or at least be backward-compatible (for example I really want a -V (--version) argument on the command line, instead of show about, which will remain but provide more info maybe).
Also, I think that if the server start feature (#200) is implemented at some point, it will be directly in rust.
I'm only a hobbyist programmer with no real experience, so i don't want assume how easy it is to implement anything (and i don't want to push goals that aren't part of the scope of the project), but it would be absolutely amazing if there was a way to download and manage modrinth modpacks via the CLI (essentially how PrismLauncher/MultiMC does).
Prismlauncher does have CLI options for doing this, but it still depends the GUI dependencies for QT6.
I was considering researching how to implement something similar to a "headless modrinth launcher", and it was actually through researching that I found this repo!
Anyway, thanks for replying, good luck!
I don't know if this is exactly what you are searching for, but there is a project called ferium you can be interested to check. This was already discussed for portablemc, and this would be a bit out of scope, but because I found ferium I abandoned the mod manager idea for PMC. Mainly because I didn't want to reinvent de wheel but also because it's interesting to have a complementary project (in rust!).
I already found Ferium yesterday, it was quite good, but a TUI PrismLauncher type experience would be ideal.
Perhaps if I learn rust more seriously I'll attempt to create one.
thanks.
I'm just thinking again of something like pmc start curseforge:<pack id> and here we go the game directly starts a modpack. No joke this would be so cool to avoid the horrible curseforge launcher! This come with many problems because it's better to have a specific work-dir for each modpack, so mods/configs don't override. And I don't know how to handle this.
you can always use prismlauncher instead of using curseforge directly you know?
Oo interesting! You know, I've been using PMC for everything for 3 years now so I'm not really aware of the GUI launcher market, thank you!
This is what i was alluding to in my earlier comment, a prismlauncher type TUI would be awesome!
Side note, thanks for 300 stars this is insane!
Perhaps this could be a dependency? https://github.com/modrinth/theseus/tree/master/theseus/src/launcher
Perhaps this could be a dependency? https://github.com/modrinth/theseus/tree/master/theseus/src/launcher
Sorry, I forgot your message... I don't think that this exactly fit what I need for the launcher, it doesn't seem to handle many corner cases that we found over the years with older versions (class path ordering) so for now I'm just going to rewrite everything, largely inspired by the Python module. I prefer doing this, because I have the feeling that we'll avoid a ton of bugs that we already solved in the history of this launcher.
It would be a huge boost to the whole community if you instead contributed that work to theseus. It would probably be easier for you too, because you'd have a starting point (rust wise). Modrinth launcher users would have a better time with legacy versions of the game, you'd have more maintenance behind the lib you're using. shrug
It would be a huge boost, I agree, but in this specific case I'm afraid that the current architecture I'm going on with would clash with their architecture. For example, they are parsing the version metadata (called "version info" in their code) ahead of time, this isn't compatible with many backward fixes PMC is doing, also they are massively using async, and I'm not a huge fan of async for this case (except for downloads!). These are just examples, but I think you can understand why I'm afraid of trying to convince people to tweak Theseus' code to fit PMC CLI needs. However, I would be really happy to contribute to Theseus, in parallel of PMC if possible, to share my knowledge (PMC code itself can be used as a documentation by the way).
Some progress update...
[ OK ] Loaded version 1.21
[ OK ] Loaded client
[ OK ] Loaded and verified 70+0 libraries
[ OK ] Loaded logger client-1.12.xml
[ OK ] Downloading... 3 MB/s 100.0% (1/1)
[ OK ] Loaded and verified 3887 assets 17
[ OK ] Downloading... 36 MB/s 100.0% (3958/3958)
This is actually insane, this download is running in a single thread with an async downloader implementation, I'm pretty sure that there is room for improvement, but this is already insanely fast. This is downloading 866 MB in 24 seconds, for full 1.21, this is not counting the JVM because it's not yet ready, and the 24 MB of the client jar file.
It would be a huge boost, I agree, but in this specific case I'm afraid that the current architecture I'm going on with would clash with their architecture. For example, they are parsing the version metadata (called "version info" in their code) ahead of time, this isn't compatible with many backward fixes PMC is doing, also they are massively using async, and I'm not a huge fan of async for this case (except for downloads!). These are just examples, but I think you can understand why I'm afraid of trying to convince people to tweak Theseus' code to fit PMC CLI needs. However, I would be really happy to contribute to Theseus, in parallel of PMC if possible, to share my knowledge (PMC code itself can be used as a documentation by the way).
And regarding this, I went back on some of my positions, mostly on ahead of time parsing, because in the end it's so much more practical.
I introduced a standardized cache (see below) to avoid polluting the minecraft directory, for now it will be used for versions and JVM manifests. Basically, this will be used for manifest files that the game don't require in its standard directories. The reasoning behind this is that I want the cache to be easily cleared, and also I wanted to avoid duplicating caches (this is currently the case, the portablemc_version_manifest.json is duplicated on each main dir, this file will be deleted by the new version). I want PMC to leave the .minecraft clean the most I can.
let mut file = dirs::cache_dir().unwrap_or(env::temp_dir());
file.push("portablemc-cache");
file.push(url_digest);Note that the URL digest is the SHA-1 of the URL.
Standard installation is now complete, now I'll be working on how to launch the game! I've added a much more complex and flexible JVM finding policy, that supports static path (with optional version checking), "System" installs finding (reading PATH, and also the standard Linux /usr/lib/jvm) and "Mojang" JVM, System and Mojang can be configured to run before one another, the default is currently (may change in the future) to search a JVM on the system and only then to install Mojang provided one, if possible. There is debugging events for each rejected candidate that doesn't match the required version.
[ OK ] Fetched version 1.21 -- 324.3 kB/s 100.0% (1/1)
[ OK ] Loaded version 1.21
[ OK ] Loaded client
[ OK ] Loaded and verified 70+0 libraries
[ OK ] Loaded logger client-1.12.xml
[ OK ] Loaded and verified 3899 assets 17 -- 2.4 MB/s 100.0% (1/1)
[ OK ] Loaded JVM version Some("21.0.3") at "C:\\Users\\theor\\AppData\\Roaming\\.minecraft_test\\jvm\\java-runtime-delta\\bin\\javaw.exe" -- 373.6 kB/s 100.0% (1/1)
[ OK ] Downloaded -- 6.0 MB/s 100.0% (4349/4349)
You can see that every download left an event, everything that is not an API, that downloads file, is now handled identically.
I would LOVE to see this converted to Rust. I'm currently using PortableMC as a backend for a Minecraft launcher I'm making with Tauri (using Rust to run the PMC Python script). It'd greatly simplify things if I could directly interact with PMC in Rust. Can't wait to see where this goes!
Thank you for the support :) this rewrite is on the right track.
How about using Go for that?
It seems to me that its far more convenient option for writing single-binary cli easy and fast.
How about using Go for that? It seems to me that its far more convenient option for writing single-binary cli easy and fast.
both go and rust produce statically linked executables, but since rust is more hipsterish, perhaps it will attract more potential contributes.
Honestly I don't know Go and I've a pretty good experience in Rust CLI so it's why I'm going with that language, not really for the potential contribute, your point is right tho.