Plans for package manager (like cargo on Rust)
funatsufumiya opened this issue · 9 comments
I think this disucussion is still early for the current development stage, are there any plans for package manager? (sorry if duplicated issue.)
I read the document about stdlib (for example path), vale would need other (FFI) libraries and build system.
If stdlib is like kind of battery included, many users can use vale as soon, but I think library eco-system is highly needed in the next stage.
Howdy!
The priority for the last ~half year has been on making a prototype for regions, since that justifies the entire language's existence. Luckily, this prototype is nearing completion (benchmarks this week hopefully, and then I'll spend a few weeks prototyping seamless concurrency to show regions' power) so I can start to turn towards other priorities soon. I think you're right in that packages/ecosystem is up near the top of that priority list.
I want something like cargo, though there's another interesting factor: cross-language support. I want to use Metacall to enable calling into other languages' libraries (something I don't think any language does yet), and I want to figure out the best way to interoperate with other languages nicely. I've heard bazel does pretty well in this regard, but haven't done much investigation.
The first step will probably be to add some flags to the compiler to point to a Rust library (.rlib) directly and ask Metacall (at compile time) what its API looks like, and have valec link it in and figure out the right way to translate / call it. It'll be an interesting quest, that's for sure.
(Also, thanks for sponsoring!)
@Verdagon
Wow, sounds pretty interesting. I always felt FFI binding is very hard. ( Nim language has interesting features in this point thanks to the transpiler to C. )
I agree with that enabling cross-language support would change this world.
If there's anything I can, I would like to help you :)
I've heard bazel does pretty well in this regard, but haven't done much investigation.
Probably this one?
https://cxx.rs/build/bazel.html
@Verdagon
Related to the above, CL-CXX-JIT maybe also interesting.
https://github.com/Islam0mar/CL-CXX-JIT
I recently think wasm can be true universal FFI as alternative of C header and DLL.
https://extism.org/
I didn't know about extism, thanks for the tip! The time for inline support is near, as we're close to done fixing the tech debt from generics and regions, so the timing is good.
Extism sounds like it could help us run arbitrary languages' code in a sandbox, which we indeed need for the "fearless FFI" feature. I'm curious how this compares to something like Metacall.
Before that, the first step will probably be interop with Rust, as they've got a nice ecosystem that Vale could benefit from. Once that's sorted, I'll take a close look at extism and Metacall (and others?) and figure out a nice direction.
@Verdagon
Metacall provides the interface for interoperation using C API, while extism is using WASM.
I think interface protocol is anything OK if it's common and maintainable, but the problem is writing interface itself by programmer would be a burden. So how easily automatically generating it would be important, like cbindgen of Rust.
Leaving aside the possible subtle differences in complexity, wouldn't WASM provide reduced performance compared to "native" strategies based on C, like Metacall does?
In other words, wouldn't be better to use Metacall instead of WASM, to guarantee better performance?
To give better context, I bring the excellent work of Frank Denis.
In January of this year, he made an update about the performance of numerous webassembly runtimes, both against javascript runtimes, as well as native code.
The graph below shows how many times slower wasm were against native code for multiple tests:
On the native side, to support my claims of wasm being slower, Frank's tests results show that wasm is about 2.32 times slower (which is not all that bad, but it is still slower). But there is a catch (see below).
Cryptographic operations scenario
Take a look at this important observation from Frank:
Two outliers have been hidden: the aegis128l and aegis256 tests. Native code takes advantage of native CPU instructions to compute AES rounds. WebAssembly, on the other hand, cannot take advantage of these instructions and has to fall back to slow, software implementations.
As a result, these AES-based tests were 80 times slower than native code when running WebAssembly. This is not representative of most applications. However, it highlights a real limitation of WebAssembly for cryptographic operations relying on AES.
Ignoring this, when using the fastest runtime, WebAssembly was only about 2.32 times slower (median) than native code with architecture-specific optimizations. Not bad!
For most use cases for wasm, this issue with AES being 80x slower isn't a concern since most of them aren't focused on encryption. But for a general purpose language (including Vale), tho, this is surely a concern.