httptoolkit/brotli-wasm

Does not work with Vite/Browser

stefnotch opened this issue · 14 comments

stefnotch commented

I tried out the example in the Readme, and after fixing it #7 and tweaking it to use TextEncoder/TextDecoder, it refused to work. At this point, I decided to inspect the imported object

import * as brotliPromise from "brotli-wasm";
console.log(brotliPromise);

To my surprise, it's basically empty.
image

Then I decided to cut to the chase and directly imported the relevant bits and pieces.

import { compress, decompress } from "brotli-wasm/pkg.bundler/brotli_wasm";

However, this fails since the WASM object doesn't seem to have been loaded yet.
image

For what it's worth, a different project of mine used Brotli-Wasm. Back then, I couldn't figure it out either and just copy-pasted the code + license. Then, I fixed it up a bit... https://github.com/stefnotch/starboard-editor/blob/d51bc17673385ad192417592add0092881379bfb/src/useCompression.ts#L1

stefnotch commented

Basically, I think the import * as wasm from './brotli_wasm_bg.wasm'; line in brotli_wasm_bg.js doesn't work in browsers.

pimterry commented

Hmmmmm, I don't know anything about Vite I'm afraid!

This project definitely does work in browsers in some configurations (I'm using it with Webpack 4 in multiple projects, and the test suite in this repo builds and tests it using Webpack 5). I'm not sure what the difference is in the Vite case though! What should happen is that:

Any idea where that's going wrong? When you say the import * as wasm doesn't work - are there any errors from there? How does Vite do bundling?

For reference the working webpack 5 config is here (there's a asyncWebAssembly feature you have to enable) and the webpack 4 is here (no WASM-specific config at all I think, it just works out of the box).

stefnotch commented

Basically, Vite uses Esbuild + some extra magic for dependencies during development. Webpack's performance is atrocious after all.
For production, Vite uses Rollup.

With the magic stuff enabled, I can't quite figure out what exactly Vite does.

With it disabled, it pretty much uses normal ES module imports.
So import * as brotliPromise from "brotli-wasm"; will simply import the index.browser.js file as an ES module.
However, index.browser.js uses the other way of exporting modules

module.exports = import("./pkg.bundler/brotli_wasm.js");

at which point the web browser goes "Uncaught ReferenceError: module is not defined" and fails.

stefnotch commented

So my next step is trying out

import * as brotliPromise from "brotli-wasm/pkg.bundler/brotli_wasm.js";
console.log(brotliPromise);

This actually correctly imports the module.
image

However, it then fails, since import * as wasm from './brotli_wasm_bg.wasm'; is basically magic. Webpack can handle it, since being able to handle anything, no matter how arcane, is Webpack's specialty.
image

Vite on the other hand can't handle it as nicely and ends up doing the following:
https://vitejs.dev/guide/features.html#webassembly
Essentially, wasm ends up being an asynchronous initialization function, which also mirrors how browsers handle WebAssembly. It's asynchronous. https://developer.mozilla.org/en-US/docs/WebAssembly/Loading_and_running

I can see a few different options:

  • Every consumer of the library has to use Webpack
  • A Vite specific hack is added
  • The browser variant uses the browser native way of loading WebAssembly https://developer.mozilla.org/en-US/docs/WebAssembly/Loading_and_running
    • I think the downside here might be that some tools will struggle to include the .wasm file in a production release. After all fetch("./brotli_wasm_bg.wasm") is slightly harder to statically analyze than the imports at the top of the file.
  • 👍1
pimterry commented

Everything inside the pkg.bundler is boilerplate wrapper output generated by https://github.com/rustwasm/wasm-pack, the standard tool for building WASM from Rust, building for a 'bundler' target (built here).

That same code is used by more or less 100% of Rust+WASM packages that exist for JS.

If that part isn't working, then it's a Vite and/or wasm-pack bug, there's not much to do here. The only custom code in this repo is the promise wrapper in index.browser.js, the Rust code, the tiny build script, and tests.

stefnotch commented

I see, thank you very much.
Apparently there is an issue over there already rustwasm/wasm-pack#1106

As for the future, apparently there is an active proposal to properly fix this at a language level
https://github.com/WebAssembly/esm-integration/tree/main/proposals/esm-integration

stefnotch commented

I guess this can then be closed as a combination of "wontfix" and "upstream issue 🐟".

pimterry commented

Ok! Hope that points you towards a solution eventually. If you do find a fix and there are small changes that can be made here to support Vite without breaking the existing setup then PRs are very welcome 😄.

In the meantime I'm actually going to leave this open - it's still a real issue regardless, and maybe this will eventually point towards some useful info for any other Vite users hitting the same problem. Even if it is an upstream issue that wasm-pack can fix, we'll need to pull through an update to actually sort it here afterwards too.

  • 👍1
kyr0 commented

@stefnotch Have you found a solution? I just bumped in the same issue(s) in the same order ;)

pimterry commented

I don't have a good answer for you either @kyr0 but I think this is a general problem with wasm-pack, and you should ping them about it, e.g. over here: rustwasm/wasm-pack#1106. I expect this probably affects all Vite users for all Wasm-Pack projects (i.e. basically every Rust-via-WASM library anywhere) so if you're keen on Vite it's well worth getting this fixed more generally. Might also be worth opening an issue with Vite too to see if there's solutions on that side.

If you do get any information on ways to work around this, or if there's any improvements on the Wasm-Pack side then do share that here, I'd be happy to fix this in brotli-wasm if possible.

  • 👍1
justin-schroeder commented

☝️ this is the way

image

  • 😄1
miroljub1995 commented

We came to the same issue. Looks like there is a way to workaround it without changing vite config:

import init, * as brotli from "../../node_modules/brotli-wasm/pkg.web/brotli_wasm";
import wasmUrl from "../../node_modules/brotli-wasm/pkg.web/brotli_wasm_bg.wasm?url";

const brotliPromise = init(wasmUrl).then(() => brotli);

?url did the trick and .wasm gets copied to output with hash, which is cool.

The only problem here is that I need to specify relative path to module. When I try as brotli-wasm/pkg.web/brotli_wasm_bg.wasm?url, I get error from vite Internal server error: Missing "./pkg.web/brotli_wasm_bg.wasm" specifier in "brotli-wasm" package

@stefnotch @pimterry I'm not bundle expert, but I think some additional items should be added to exports along with the current?

"exports": {
    ".": {
      "import": "./index.web.js",
      "browser": "./index.browser.js",
      "require": "./index.node.js",
      "default": "./index.web.js"
    }
  }

After that it would be nice to use it like, or something better:

import init, * as brotli from "brotli-wasm/brotli_wasm";
import wasmUrl from "brotli-wasm/brotli_wasm_bg.wasm?url";

const brotliPromise = init(wasmUrl).then(() => brotli);