discussion for moving the rust on android ecosystem forward
dvc94ch opened this issue · 32 comments
so currently a lot of the core components are spread out among various maintainers. it's hit @MarijnS95 the hardest since he is still maintaining legacy code which unless a company explicitly requires and pays for this doesn't make any sense what so ever. my suggestion would be this:
- create a new org
- move
rust-windowing/android-ndk-rs,rib/android-activity,cloudpeers/xbuildinto new org rm -rf ndk-glue ndk-macro ndk-build cargo-apk
to move in this direction I think that @rib @MarijnS95 @msiglreith would need to agree that this is the way forward. I'm maintaining xbuild as part of my day job, but it's unlikely I'll be able to contribute much to android-ndk-rs and android-activity.
Thanks for starting this discussion.
I had recently been wondering if we could modularize the android-ndk-rs repo via git filter-branch and then as suggested here it could then make sense to put those repos under a new org.
That way we could perhaps "archive" the monolithic repo. In the case of ndk-glue / ndk-macros I suppose it could be a separate question whether to split those out and then immediately archive or else could simply end up archived as part of the monolithic repo.
For cargo apk and ndk-build maybe that's more a question for you and @MarijnS95 than me. Splitting them out as separate repos seems like it would make sense to me though.
since it sounded like there was some agreement on the idea (from the other threads too that lead us to this issue) I think for starters I'll take a stab at filter branching android-ndk-rs and put those repos up here https://github.com/Android-Rust if that vaguely works and can invite you all as co-owners if that looks like a viable direction
As I said I'm mostly interested in maintaining xbuild as the defacto tool for building and deploying cross platform rust applications. The main blocker so far was no real rust gui framework. But for now I think dioxus is good enough and in a few years glacier/piet-gpu/xilem will probably be the way to go. I can continue maintaining it in the cloudpeers repo for now, as it seems that @MarijnS95 and @rib only care about android at the moment. Yeah there is a lot of experimental code in xbuild, but the android and iOS parts are getting thoroughly tested and used and improved.
I wonder if an organisation that was framed more like "Mobile Rust" might be more practical than one that is android specific? considering how common it would be to want to make crates that are useful cross-platform?
Okey for reference it was pretty straightforward to filter out each android-ndk-rs project, and initial repos can be seen here: https://github.com/Android-Rust
This is how I filtered them all:
git-filter-repo
(nicer than using git filter-branch)
git clone https://github.com/newren/git-filter-repo
ndk-sys
(takes into account that the code originally moved from android-ndk-sys/)
git clone https://github.com/rust-windowing/android-ndk-rs.git ndk-sys
cd ndk-sys
py ../git-filter-repo/git-filter-repo --subdirectory-filter ndk-sys --subdirectory-filter android-ndk-sys
ndk
(takes into account that the code originally moved from android-ndk/)
git clone https://github.com/rust-windowing/android-ndk-rs.git ndk
cd ndk
py ../git-filter-repo/git-filter-repo --subdirectory-filter ndk --subdirectory-filter android-ndk
cargo apk
git clone https://github.com/rust-windowing/android-ndk-rs.git cargo-apk
cd cargo-apk
py ../git-filter-repo/git-filter-repo --subdirectory-filter cargo-apk
ndk-build
git clone https://github.com/rust-windowing/android-ndk-rs.git ndk-build
cd ndk-build
py ../git-filter-repo/git-filter-repo --subdirectory-filter ndk-build
ndk-glue
git clone https://github.com/rust-windowing/android-ndk-rs.git ndk-glue
cd ndk-glue
py ../git-filter-repo/git-filter-repo --subdirectory-filter ndk-glue
ndk-macro
git clone https://github.com/rust-windowing/android-ndk-rs.git ndk-macro
cd ndk-macro
py ../git-filter-repo/git-filter-repo --subdirectory-filter ndk-macro
ndk-context
git clone https://github.com/rust-windowing/android-ndk-rs.git ndk-context
cd ndk-context
py ../git-filter-repo/git-filter-repo --subdirectory-filter ndk-context
@rib heh I intended to name it Rust-Android to put emphasis on Rust :)
I'm fine to start and drop ndk-glue and ndk-macros now that winit migrated to @rib's crate. Not sure if we need to publish a new crate with a big DEPRECATED marker at the top and/or #[deprecated] annotations the moment winit 0.28 releases, I'll leave that to @rib. Again, I think most ndk-glue users are using it in conjunction with winit, and having active markers here might help otherwise-unaware users (those that don't read changelogs or happen to be copypasting an older example) discover that winit changed crates entirely.
More importantly for me however: we should nuke ndk-context. We discussed many times before that statics are error-prone by design and extermely annoying to use. At the same time they are (currently) representing an Activity which some crates may have come to rely on, blocking support for multiple Activitys unless we start storing getApplicationContext(). I originally suggested that the "main" function should receive an object representing its Activity and passing that on to all crates needing it, solving all those problems. This is effectively what @rib ended up implementing in his android-activity crate, together with winits AndroidApp::with_android_app(). I suggest we force all crates to use a similar flow (perhaps with ndk-context exclusively hosting struct AndroidContext for conciseness), as that would have also helped us solve the following app_dirs2 issues more cleanly:
#126
app-dirs-rs/app_dirs2#33
tectonic-typesetting/tectonic#926
okey, I've sent owner invites for that org, and I guess we can decide whether that seems like the right direction vs e.g. leaving the repos under the rust-windowing org or not splitting these out etc
@rib heh I intended to name it
Rust-Androidto put emphasis on Rust :)
Yeah we can rename. I was also pondering the more general "Mobile Rust" or "Rust Mobile".
(we can also just delete crates like ndk-glue / ndk-macro if we figure it's fine that they are archived in the original repo)
Do other people feel this way about ndk-context? I don't think the webbrowser crate would have been possible without it [0]. I consider it a major success story.
in my opinion you should drop ndk-glue and ndk-macro. the idea is coming up with a plan for the future not maintaining backwards compatibility. anyone using rust on mobile must understand that it's experimental.
but yeah extending ndk-context to multiple activities means that any activity that wants to use rust needs to update ndk-context onCreate. currently we panic if we overwrite an existing ptr. I now think that's a mistake.
(I cannot give detailed responses now, expect more detail in an hour)
@dvc94ch your link doesn't render correctly.
@rib I'd still prefer to bundle relevant repos together, if possible. Ndk+ndk-sys, and cargo-apk+ndk-build. Ndk-glue and ndk-macros can remain here to get their deprecation marks and be ultimately removed.
Not sure how to deal with open PRs though; merge them here and bring the result to the new org?
For starters we can also transfer the entire repo to the new org and leave the split for some time later.
agree with @MarijnS95. would be a shame to loose all these stars. I think ndk+ndk-sys+ndk-context+ndk-examples should stay in this repo and we should transfer this one. those are the only parts I still care about. I guess before we do anything we need to answer these questions:
- can we stop maintaining ndk-glue/ndk-macro?
- what build system are we going to recommend to new users? gradle (android power user)/cargo-apk (rust/android game dev)/xbuild (rust cross platform application dev)
- what's the future of
ndk-context, is it really so bad to have a global variable?
(Am on mobile atm so quick replies)
Ndk-sys combined with ndk makes sense yeah.
I was wondering if ndk-build being split out could make it more inviting for others to use though perhaps? E.g. there was once a suggestion that it could be shared by cargo ndk. If it's really more of an internal dep for cargo apk though then keeping combined makes sense.
With ndk-context that crate has been a really useful solution for decoupling library creates than need to use JNi from the glue crate. It's true that it still imposes limits atm on supporting multiple activities but I'm not sure of a good solution that can keep libraries decoupled from the glue, or having to make assumptions about running as part of a standalone native app vs being used as part of an embedded library. Maybe ndk context needs to support attaching the Activity to thread local storage somehow?
Do other people feel this way about
ndk-context? I don't think the webbrowser crate would have been possible without it [0]. I consider it a major success story.
@dvc94ch if you write the link label like [1]: https://github.com/amodm/webbrowser-rs/blob/main/src/android.rs it'll resolve. But then a blue 1 shows up in the text. You might also label a bit of text using the same label ([a bit of text][1]), or even make the text the label ([the text the label], with [the text the label]: ...).
Yes, this may be a success story, but we'd have to pass a different activity/context to it somehow. Funnily enough your example link shows a very similar workaround for Termux when compiling for Android but no Activity pointer is available 😉
agree with @MarijnS95. would be a shame to loose all these stars. I think ndk+ndk-sys+ndk-context+ndk-examples should stay in this repo and we should transfer this one. those are the only parts I still care about. I guess before we do anything we need to answer these questions:
Right, don't forget about the stars, watchers, and open and past issues+PRs.
- can we stop maintaining ndk-glue/ndk-macro?
android-activity has already superseded it - I don't want to maintain it now that @rib implemented a backend purely in Rust which is what I initially intended ndk-glue to become (with aforementioned design principles so that it doesn't suffer from all the issues).
- what build system are we going to recommend to new users? gradle (android power user)/cargo-apk (rust/android game dev)/xbuild (rust cross platform application dev)
I'd say this depends on their usecase:
- (Android) app purely in Rust:
xbuild, and we deprecatecargo-apkwhen I've ported all the missing features over? - "Some" Rust code, or a full Rust activity, as part of a bigger app: Mozilla's Gradle plugin, or
cargo ndkif users wish to hack it together themselves. - Any other use-case?
These borders can muddy over time when more combinations start to be supported by e.g. xbuild, if someone needs it and wishes to put the work in.
- what's the future of
ndk-context, is it really so bad to have a global variable?
Perhaps just host the AndroidContext struct? We just highlighted why the static global is annoying; this becomes prevalent when switching a fullscreen Activity to a windowed resizable (pop-out) activity, where Android relaunches a new Activity inside the same process breaking all the asserts along the way.
Ndk-sys combined with ndk makes sense yeah.
👍
I was wondering if
ndk-buildbeing split out could make it more inviting for others to use though perhaps? E.g. there was once a suggestion that it could be shared bycargo ndk. If it's really more of an internal dep for cargo apk though then keeping combined makes sense.
Not sure if this really matters. The crate has always been published to crates.io and could be used by anyone, it being developed in more-or-less monorepo style shouldn't be that big of a barrier for contributions - but it does make it more inviting to apply breaking changes to ndk-build.
It was always the idea to make cargo ndk use it so that it doesn't lag behind so much in light of NDK updates, and have a more shared codebase with more closely focused developer resources. Unfortunately the author told me it would be a welcome addition, but wouldn't happen unless I (or someone else) were to put the work in.
At the same time I might've liked to integrate it as part of the monorepo for the same reasons, but haven't been in need of a cargo-ndk2 yet 😉
With ndk-context that crate has been a really useful solution for decoupling library creates than need to use JNi from the glue crate. It's true that it still imposes limits atm on supporting multiple activities but I'm not sure of a good solution that can keep libraries decoupled from the glue, or having to make assumptions about running as part of a standalone native app vs being used as part of an embedded library. Maybe ndk context needs to support attaching the Activity to thread local storage somehow?
This is always a problem; whether to arrange inter-crate communication behind a users back and simplify for the generic case while limiting the "more niche" / advanced uses (though I wouldn't call restarting a second Activity within the same process/application "niche"), or expose everyone to a few extra lines of code, making things explicit and solving many bugs along the way.
EDIT: Fwiw an Android Activity written in Rust always has a clear entrypoint where the VM and Context are handed over.
I feel like preferring the latter.
agree with @MarijnS95. would be a shame to loose all these stars. I think ndk+ndk-sys+ndk-context+ndk-examples should stay in this repo and we should transfer this one. those are the only parts I still care about. I guess before we do anything we need to answer these questions:
re: stars and issues yeah the original repo should be kept for at least the ndk + ndk-sys crates at least for continuity
I tend to think it would still make sense to split out ndk-context (it's not tightly coupled with anything else and it also doesn't have a long history or backlog of issues that would give cause to keeping it coupled.
I'm not quite sure what to do with the ndk-examples. There is a pretty decent collection of examples associated with android-activity here: https://github.com/rib/android-activity/tree/main/examples and maybe those could be split out into a replacement android-examples repo and put here?
- can we stop maintaining ndk-glue/ndk-macro?
I would tend to think so but I think that's really up to @MarijnS95 at the moment.
- what build system are we going to recommend to new users? gradle (android power user)/cargo-apk (rust/android game dev)/xbuild (rust cross platform application dev)
Personally I'm not one for wanting to pick sides / winners when it comes to mobile / android build tools for Rust projects. There are so many use cases and trade offs and levels of maturity etc.
I have my own personal preference atm for using cargo ndk + Gradle but also ok with trying to facilitate others experimenting with alternatives. I think it's better to promote experimentation and maybe sign post a bit with a description of pros and cons for different tools.
In my view, if there is an organisation for collecting android/mobile rust crates it'd be fine to have overlapping scope between projects (e.g. build tools) and the benefit of the organisation can be more about facilitating discovery and cross-pollination than anything else.
- what's the future of
ndk-context, is it really so bad to have a global variable?
I think we should keep it for now - it solves a real problem, even though it's not perfect. The issue with having a global activity is a valid concern for supporting multiple activities in a single process (which Android allows) but we should treat that as a separate issue.
Just to add a bit more about ndk-context: I do generally agree with @MarijnS95 suggestion earlier that projects could probably be encouraged to support explicit initialization with the state they need (similar to Winit's with_android_app()) - I suppose I'm just thinking that that's a much longer term goal and so for now I think ndk-context is a useful crate to maintain considering the big improvement over the previous situation where library crates were depending on the glue crate.
So decisions:
- ndk-glue and ndk-macro are rm -rf
- ndk-context is moved into it's own repo and kept until a viable alternative emerges
- examples are moved and consolidated into their own repo, maybe we can add a book/docs to that repo too
- cargo-apk/ndk-build is moved to a separate repo and maintained but we start promoting xbuild in examples and docs. Once the external library feature is added we can discuss deprecating cargo-apk
Is this a fair summary of the conversation?
No denying that ndk-context is already miles better than crates depending on any sort of glue layer directly.
with the filter branches I ended up redoing them so they would keep things like the license files + .github .gitignore + rustfmt
I still re-ran the filter branch for ndk-glue and ndk-macro and marked the repos as archived to see how that could work for keeping these easily accessible for posterity (but we can still delete these if we prefer)
(essentially the main two filtered repos left are cargo-apk (which now keeps ndk-build in the same repo) and ndk-context (which now keeps the license files etc as mentioned)
In terms of archiving ndk-glue and ndk-macros what do we want to do here exactly? Everything except ndk and ndk-sys will be removed from the original repo and it seems a shame if both of those repos are simply consigned to the history of the ndk repo instead of having an archived repo where they remain easily accessible?
For the examples; the easiest thing with these would be for me to split out the example from android-activity instead of keeping the originals just because the current examples assume using ndk-glue
Is there a strong interest in having the looper and jni examples specifically (they can probably be ported if so - just not sure if the looper one might be awkward)
To test I have split out the examples from android-activity here: https://github.com/Android-Rust/rust-android-examples
@dvc94ch with the way xbuild works would it be possible to have parallel support for building those examples while still maintaining the current ability to build with cargo ndk and Gradle?
Like with the NativeActivity examples the README shows how to build them with cargo apk or cargo ndk + Gradle - it could be good to support building with xbuild too - which would also be able to build the GameActivity examples.
I was going to rename the organisation linked above to "Rust Mobile" to have a more inclusive scope if no objections to that?
Not really a problem. The instructions for cargo-apk work almost identically with xbuild if they don't require anything special in the manifest. The largest difference is specifying the android manifest in manifest.yaml instead of trying to use some contortion to express it in the Cargo.toml
That sounds good; so then it'd be possible to show how people can opt in to using xbuild if they want without precluding other options too. I still haven't had a chance to look at xbuild properly but I think I'd be more inclined to give it a try if I knew it was compatible with a project using Gradle - so worst case it'd be easy to switch back to Gradle if necessary.
I think I'll re-filter the cargo-apk repo to keep the examples (at least the hello world example) since that's effectively needed for the cargo-apk CI and it'd be awkward pulling in an external example for the CI
Ok the updated organisation url is now: https://github.com/orgs/rust-mobile (though the old one redirects)
btw for reference I took an initial look at getting cargo-apk building via CI in the standalone repo: rust-mobile/cargo-apk#1
that PR also includes a port of the examples to android-activity.
If we do commit to this direction we'll have to decide on the precise ordering of things. E.g. there are a few pending PRs atm for cargo apk and so there would need to be a decision about whether to merge those first and then re-filter out cargo-apk or else those PRs would need to effectively get merged with the standalone repo.
Also for reference here; the latest filter-repo for cargo-apk was done like this:
py ../git-filter-repo/git-filter-repo --path cargo-apk --path ndk-build --path ndk-examples --path-rename ndk-examples:examples --path LICENSE-APACHE --path LICENSE-MIT --path Cargo.toml --path rustfmt.toml --path .github --path .gitignore
done
Yup, ramping down cargo-apk development too now that all my changes landed in xbuild 🎉