Generate a single JavaScript file

Question

Generate a single JavaScript file

kellytk opened this issue 5 years ago · 18 comments

kellytk commented 5 years ago

💡 Feature description

It would be very useful for wasm-pack to provide an option to generate a single amalgamated JavaScript file.

Answer 1 · 2019-08-13T08:01:35.000Z

@kellytk What is your use case? Wasm-pack currently supports the following targets:

bundler which is intended to be used with Webpack (so it's up to Webpack to bundle everything into a single file)
nodejs which is intended to be run in Node. it uses require to auto-load the files, which is normal in Node.
web which is intended to be run directly in modern browsers, the browser will automatically load the files using import, which is normal.
no-modules which is intended to be used everywhere, it already outputs a single file.

Answer 2 · 2019-08-13T08:35:13.000Z

Option 4 may be what I want however I can't get it to run successfully.

When I run wasm-pack build --target no-modules on https://github.com/kellytk/yew-wasm-pack-minimal/tree/initial-commit it generates:

[INFO]: 🎯 Checking for the Wasm target...
[INFO]: 🌀 Compiling to Wasm...
Finished release [optimized] target(s) in 0.04s
[INFO]: ⬇️ Installing wasm-bindgen...
error: failed to generate bindings for JS import __cargo_web_snippet_a1f43b583e011a9bbeae64030b81f677e6c29005
caused by: local JS snippets are not supported with --target no-modules; use --target web or no flag instead
Error: Running the wasm-bindgen CLI
Caused by: failed to execute wasm-bindgen: exited with exit code: 1

Do you know what mistake I'm making?

Answer 3 · 2019-08-13T12:52:39.000Z

@kellytk That's because inline_js isn't currently supported with the nodejs or no-modules targets. It is intended for them to be supported, but it hasn't been done yet: rustwasm/wasm-bindgen#1525

Answer 4 · 2019-08-22T16:01:01.000Z

I've got a use case where my final deliverable is a single html file. Currently I'm using parcel to inline all the js and css. If wasm-pack could generate a single js along with the ts declarations it would work flawlessly for what I need.

Answer 5 · 2019-08-23T02:56:02.000Z

@EliSnow I made https://github.com/yewstack/yew-wasm-pack-minimal and used Rollup. I'd gladly welcome a Parcel-based solution if you're so inclined.

Answer 6 · 2020-03-26T11:33:34.000Z

I tried to use wasm-pack with rollup but did not manage to create a minimal example.
A detailed report is over at rollup/plugins#269 with an example repository for reproducing the issue.

After reading https://rustwasm.github.io/docs/wasm-pack/commands/build.html?highlight=bundler#target I thought that wasm-pack is supposed to work with different bundlers, not just webpack but as @Pauan mentioned this might not be the case? It would be helpfull to specify which requirement the bundler has to fulfill.
(I have the impression that import await needs to be supported? On the other hand parcel (or specifically parcel-plugin-wasm.rs) seems to handle it yet differently https://github.com/rustwasm/rust-parcel-template/blob/master/js/index.js by supporting imports from the Cargo.toml (?))

Answer 7 · 2020-03-26T19:29:51.000Z

@benmkw Webpack is the only bundler that wasm-pack natively supports. This is because Webpack supports importing .wasm files directly, whereas other bundlers don't.

However, even though wasm-pack doesn't natively support other bundlers, you can still use them with plugins. For example, you can use the @wasm-tool/rollup-plugin-rust plugin with Rollup (which works really well).

Answer 8 · 2020-03-26T22:50:50.000Z

For example, you can use the @wasm-tool/rollup-plugin-rust plugin with Rollup (which works really well).

I somehow managed to look in all the wrong places (first here then here) and missed that one, thanks a lot I'll give it a go! 😃

Answer 9 · 2020-03-27T13:12:06.000Z

Sorry I was a bit quick/ tired to respond, it appears that https://github.com/wasm-tool/rollup-plugin-rust is the same as https://github.com/rollup/plugins/tree/master/packages/wasm in that it needs all code that interacts with wasm to be async.

As far as I see this is a big drawback compared to the approach by webpack https://github.com/webpack/webpack/tree/master/examples/wasm-simple because it makes it much harder to use wasm functions as callbacks/ simple functions together with the dom.

It is not clear to me how to structure the program in face of async wasm loading such that it does not leak all over the code so this makes these two plugins hard to use for me.

I may want to avoid reloading the same module (without relying on correct browser and server configuration) and persist state in the same wasm module over multiple calls to it, both of which is harder using the async api not handled by the bundler.

The benefit of exposing the async nature of wasm to the programmer seems to me that it makes it easier to avoid the page from blocking but the page may only usefull with the wasm loaded anyway.

(I also probably made a mistake while I tried https://github.com/wasm-tool/rollup-plugin-rust because the wasm file is not served correctly cause its in the wrong dir but the example was not using rust from js but just using rust intead of js so this may be why it was unclear https://github.com/benmkw/svelte_rust_test)

Back to my first investigaton of bundlers, my hope was to get something like:

import some_fn from "some_package"
// just use some_fn

Possibly at the expense of a little more loading time upfront.

This appears to be only possible using webpack/ it would be interesting if/ when/ how this would be possible in other bundlers/ if this is a supported approach or discouraged for other reasons (and what the suggested alternative for interacting with non async code is)

Answer 10 · 2020-03-27T16:56:25.000Z

@benmkw in that it needs all code that interacts with wasm to be async.

That's not because of wasm-pack (or the plugins), it's fundamentally how Wasm works. All of the Wasm APIs (such as instantiateStreaming) are asynchronous.

Until browsers get native .wasm support, you simply must deal with the fundamental asyncness of Wasm. There is no way around that.

As far as I see this is a big drawback compared to the approach by webpack

Note that Webpack also requires async: you must use dynamic import() (which returns a Promise) in order to import .wasm files. You cannot import .wasm files synchronously, not even with Webpack.

Bundlers and plugins cannot make Wasm synchronous, because the browser itself requires Wasm to be loaded asynchronously.

my hope was to get something like import some_fn from "some_package"

That is impossible until browsers get native .wasm support. Not even Webpack supports that, you have to use dynamic import() with Webpack.

how this would be possible in other bundlers

We can't really do anything about that: if you want Webpack-style importing of .wasm files, you'll need to ask your favorite bundler to add in that feature.

The benefit of exposing the async nature of wasm to the programmer seems to me that it makes it easier to avoid the page from blocking but the page may only usefull with the wasm loaded anyway.

No, the "benefit" of making it async is that the browser forces us to make it async. We have no choice. The browser simply does not allow you to load Wasm synchronously.

I may want to avoid reloading the same module (without relying on correct browser and server configuration) and persist state in the same wasm module over multiple calls to it, both of which is harder using the async api not handled by the bundler.

This is very easy to accomplish, since ES6 modules are never reloaded:

// wrapper.js
import wasm from "./path/to/Cargo.toml";

export default wasm();

Now you can import that ES6 module and it's guaranteed to only load the Wasm file once:

import wasm from "./path/to/wrapper.js";

async function foo() {
    let exports = await wasm;
}

It is not clear to me how to structure the program in face of async wasm loading such that it does not leak all over the code so this makes these two plugins hard to use for me.

Normally you would structure your app like this:

async function main() {
    let wasm = await loadWasmAsyncSomehow();

    // The rest of your app's initialization code
}

main().catch(console.error);

In other words, your app's main entry point is async. It will asynchronously load the Wasm, and then afterwards everything else (including the DOM) can be fully synchronous.

So you only need a single async function, and you only need to deal with the asyncness in one place (it does not leak throughout your app). This also guarantees that the Wasm is only loaded once.

For libraries it is trickier, but your library can export a function which initializes the library, and then you leave it up to the application to call the init function:

// Library
export default async function () {
    let wasm = await loadWasmAsyncSomehow();

    return {
        // Your library's API goes here...
    };
}

// Application
import init from "some-library";

async function main() {
    let library = await init();

    // Rest of app code...
}

main().catch(console.error);

Answer 11 · 2020-03-27T17:21:50.000Z

P.S. You can combine the async main approach with the wrapper approach, which actually gives you the exact same behavior as Webpack:

// wrapper.js
import wasm from "./Cargo.toml";

export let exports = null;

export async function init() {
    exports = await wasm();
}

// some-file.js
import { exports } from "./wrapper.js";

// You can use exports synchronously!

// main.js
import { init } from "./wrapper.js";

async function main() {
    await init();

    let someFile = await import("./some-file.js");

    // Run rest of initialization code
}

main().catch(console.error);

Now you can import and use the Wasm exports synchronously, as long as you make sure that your application is initialized after init() finishes (this is guaranteed by the async main function).

You can compare that to the Webpack approach:

// some-file.js
import * as exports from "./foo.wasm";

// You can use exports synchronously!

// main.js
async function main() {
    let someFile = await import("./some-file.js");

    // Run rest of initialization code
}

main().catch(console.error);

As you can see, it's very similar, the only real difference is that you needed to create a small wrapper file.

The reason why the Rollup plugin doesn't create this wrapper file for you is because it's very inflexible: what if you want to initialize the Wasm multiple times? There are use cases for that.

Answer 12 · 2020-03-27T18:22:41.000Z

Thanks for your explanations.
I now managed to piece it together with the svelte framework.

<script>
	import { onMount } from "svelte";
	import wasm from './../../rust_wasm/Cargo.toml';

	let add = null;

	onMount(async () => {
    	  let wasmer = await wasm();
	  add = wasmer.add;
	});
</script>

<main>
	<!-- does work -->
	<p> {add ? add(1,2) : "not yet" } </p>

	<!-- does not work because onMount is called after first render-->
	<!-- <p> {add(1,2)} </p> -->

</main>

Note that Webpack also requires async: you must use dynamic import() (which returns a Promise) in order to import .wasm files. You cannot import .wasm files synchronously, not even with Webpack.

Apparently in webpack you can now do

import await { add as mathAdd, factorial, factorialJavascript, fibonacci, fibonacciJavascript } from "./math";

console.log(add(22, 2200));

so the import sort of blocks until the promise resolves and thus the rest of the file can treat the function from the async import as if they are coming from a non async import. (I have not tried it actually but as far as I see)

This is very easy to accomplish, since ES6 modules are never reloaded:

That's good to know!

In other words, your app's main entry point is async. It will asynchronously load the Wasm, and then afterwards everything else (including the DOM) can be fully synchronous.

yeah that's kind of what I meant with leak (but viewed this way its not really the right word), in the sense that you have to put it all into the function and can't communicate with the outside (or have to use the thing where you put a dummy value in and then hope the async code resolves quickly as I did in the example now)

Sorry for derailing this issue 😃

Answer 13 · 2020-03-27T19:09:33.000Z

@benmkw Apparently in webpack you can now do

That is an experimental syntax which is not enabled by default. And it will likely never be standardized. It also "taints" the module, so now you need to import that module with import await, all the way to the root module.

The top level await proposal will fix this problem, but it's not implemented in any browsers.

In any case, that's Webpack specific, so you would need to ask other bundlers to implement it as well.

or have to use the thing where you put a dummy value in and then hope the async code resolves quickly as I did in the example now

In the above post I showed a way to get the exact same behavior as Webpack: there are no race conditions, and it is guaranteed that you can fully access the Wasm synchronously inside of your rendering code, without any hacks.

Answer 14 · 2020-03-27T20:04:07.000Z

you can fully access the Wasm synchronously inside of your rendering code, without any hacks.

Basically I did the same as they did for react https://www.telerik.com/blogs/using-webassembly-with-react here but upon closer inspection how others deal with this I also found https://github.com/BrockReece/vue-wasm which is what you mean (I hope) and it works in svelte as well, I was just to focussed to see the bigger picture.

// main.js
import App from './App.svelte';
import wasm from './../../rust_wasm/Cargo.toml';


const init = async () => {
    const wasmer = await wasm();
    const w_add = wasmer.add;

    const app = new App({
        target: document.body,
        props: {
            add: w_add
        }
    });

};

init();

// App.svelte
<script>
	export let add; // the wasm function
</script>

<main>
	<p> {add(1,2)} </p>
</main>=

Thanks for insisting that my solution was not proper 👍

Answer 15 · 2020-09-02T12:27:40.000Z

https://rustwasm.github.io/wasm-bindgen/examples/without-a-bundler.html mentions a number of limitations with the no-modules option. It would be good to have this to use with web-view projects that involve the use of javascript snippets. I'm currently hacking this myself with regexes find/replace in the web output.

Answer 16 · 2023-11-10T19:13:09.000Z

@kellytk What is your use case? Wasm-pack currently supports the following targets:

1. `bundler` which is intended to be used with Webpack (so it's up to Webpack to bundle everything into a single file)

2. `nodejs` which is intended to be run in Node. it uses `require` to auto-load the files, which is normal in Node.

3. `web` which is intended to be run directly in modern browsers, the browser will automatically load the files using `import`, which is normal.

4. `no-modules` which is intended to be used everywhere, it already outputs a single file.

It would be much much more easier if we could bundle wasm binary into js file. In many cases, we cannot load resources from node_modules directly, but we can import the package which only contains js files. The default init function will not necessarily work, especially when you have many different bundle methods, so many configure options...

Edit

For people who are looking for solutions, checkout this post

Answer 17 · 2024-03-20T22:15:35.000Z

Due to this limitation, js-tiktoken seems to have hacked together it's own solution so it can create an hybrid package that can be imported from both node and bundlers using package.json exports.

See https://github.com/dqbd/tiktoken/blob/main/wasm/scripts/postprocess.ts

Answer 18 · 2024-05-01T09:21:35.000Z

I'm guessing that what people need is something that can be used on both web and node platforms with the same api.
Currently, nodejs is sync, the web is async, and the wasm file is separate from the js, which will lead to a lot of problems when packaging and distribution. For some scenarios, we are willing to trade the size of base64 for ease of use.

const base64 = "wasm_base64"

export default async function (env: Env = {}) {

  const buffer = decode(base64);

  const wasmModule = await WebAssembly.compile(buffer)
  const instance = (await WebAssembly.instantiate(wasmModule, env)) as Instance

  function fib(n: number): number {
    const fn = instance.exports.fib
    return fn(n)
  }

  return {
    instance,
    fib
  }
}