Evaluate multi-module use cases

Question

Evaluate multi-module use cases

atsteffen opened this issue 2 years ago · 2 comments

Seems like use cases where you want to separate out shared functionality into a library and use it in multiple other libraries or binaries will inevitably come up.

In the simple case, you could have:

multiple crates in a single maven project

In the more complex (and more interesting) case, you could have:
2) multiple crates spread across maven projects with dependencies between them

How do you go about adding dependencies between rust artifacts produced in separate maven projects? Is this trivial with built-in cargo functionality? Is this a bad idea?

In other words, can you drive a complex, multi-module, mixed language project, that includes rust, entirely using maven (in the presence of cargo)? This closely matches the use-case for our very large, multi-module java/web application, which currently pulls in c++ binaries and system libraries in a rather convoluted way. We can accomplish what we need using various tools, but baking more of this into our pom.xml files greatly simplifies the surface area developers need to interface with, and ups the likelihood that all dependencies are up-to-date in the final product.

Related Note:
Incremental building capabilities in maven are actually really essential this type of maven first build pipeline: https://maven.apache.org/extensions/maven-build-cache-extension/index.html

I can start looking into this, but wanted to verify it's validity first since my rust experience is rather limited.

Answer 1 · 2023-01-26T07:56:44.000Z

I think that this is a no-op:
Rust crates can already do this from rust-maven-plugin's point of view: Rust dependencies are generally rebuilt all from source.

Shared code would just be built as regular libs without Maven integration.

Any open source shared libs you can upload them crates.io, otherwise you can:

Refer to them by path.
Load them from a git repo.
Start your new registry.

See: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html

As for caching across builds, see:
https://doc.rust-lang.org/cargo/guide/build-cache.html

As that page mentions, there's also https://github.com/mozilla/sccache, but you're only likely to need this once your volume of code starts getting large.

If there are specific features (e.g. customizing --target-dir) that come off the back of your needs we can re-evaluate, but for now I think we can close this issue as I think that all dependency matters can be dealt with from within Cargo.

Answer 2 · 2023-01-31T15:31:54.000Z

Thanks for the feedback. I agree there is no action item yet. I will test various approaches using built-in cargo functionality.

Where projA has a set of library functionality in lib.rs, that is used by rust binaries or dylibs in projB and projC.
I should be able to have java integration tests in projB and projC that are affected by changes in the lib.rs in projA. I should be able to incrementally change lib.rs and run all tests by invoking: mvn test from the root (projP). Ideally, this could be done without having to increment lib.rs version and dependency references in projB and projC. It would also be nice not to have to rely on relative paths (in case modules move), or the installation of additional tools. At least in this case, I'm trying to keep the rust source fully embedded (possibly hidden) in the java project, and not treated like an external dependency.

Big picture:
As a java developer, I think rust + java is a match made in heaven (especially when you add in maven/maven central and cargo/crates.io). They both default to memory safety (although by different means). Java is historically strong in web servers, while rust has wasm-pack. Java is rich in libraries for high level application orchestration, while rust can be optimized for low level performance operations. JNI exists and is supported in rust, etc.

Modularity is key to maintaining long-lasting software projects. Multi-module maven projects are extremely common these days ("Almost 75% of Maven users work with multi-module projects" link). Making a use case like the one described above seamless would really make this plugin compelling IMHO.

Sorry if this is not the right place to have high-level discussions like this. I will certainly take point on any discovery and implementation needed related to this request as I have time over the coming weeks.