Bundling issues with Snowpack

Question

Bundling issues with Snowpack

basham opened this issue 3 years ago · 12 comments

Earthstar suggests importing the browser bundle from a CDN.

<script type="module">
  import * as Earthstar from "https://cdn.earthstar-project.org/js/earthstar.bundle.v7.1.0.js";
</script>

However, I would like to install the bundle locally, so my build environment (Snowpack) can include it and my PWA library (Workbox) can cache it.

Could you please add the browser bundle to the npm package?

In the meantime, I am solving it by referencing a custom repo that includes the bundle, in package.json.

{
  "dependencies": {
    "earthstar": "basham/earthstar-bundle#v7.1.0"
  }
}

Answer 1 · 2022-02-28T10:40:15.000Z

I saw this too when trying to build something! Sorry for the inconvenience, this will be done very soon.

Answer 2 · 2022-03-02T16:14:27.000Z

Hey. I've publish v8.0.0 to NPM. I need to add this to the docs, but you can import browser / node specific stuff this way:

import { Replica } from "earthstar";
import { ReplicaDriverIndexedDB } from "earthstar/browser";

Answer 3 · 2022-03-02T16:30:39.000Z

The import path changes are nice. I was doing this in one part of my code:

import { ReplicaDriverSqlite } from 'earthstar/src/entries/node.js'

It'll be nice doing this:

import { ReplicaDriverSqlite } from 'earthstar/node'

However, I'm not seeing the bundle in the npm package.

https://unpkg.com/browse/earthstar@8.0.0/

Without the pre-made bundle, I will need to use node polyfills in Snowpack to make it work. Because the "universal" Earthstar code still depends on things like "buffer" and other such things. (I'd have to work on finding the full list of import issues, if you'd find that valuable.)

Answer 4 · 2022-03-02T18:58:05.000Z

This is valuable, as I have not been able to test all these environments! A big part of reworking earthstar was to put all the node specific stuff in its own compartment, so we definitely don't want Buffer sneaking in via the universal or browser imports. Do you get any stack traces for what's importing it?

Answer 5 · 2022-03-02T21:34:56.000Z

I know I encountered problems with buffer with either Earthstar v6 or v7. But I'm not seeing that particular issue now.

After some more investigation, the problem is more complex than I hoped. There are many direct and indirect references to earthstar/esm/deps.js from modules that are referenced in both the universal and browser entries. As such, all of these dependencies are trying to be converted to work in the browser:

chalk
fast-deep-equal
fast-json-stable-stringify
rfc4648
sha256-uint8array
@noble/ed25519
concurrency-friends
earthstar-streaming-rpc
earthstar-streaming-rpc/browser

Some of these do not compile cleanly in Snowpack (such as chalk).

Some of these compile but error in the browser. This browser error suggests some indirect dependency:

SyntaxError: The requested module '/_snowpack/pkg/@sgwilym.urlpattern-polyfill.v1.0.0-rc8.js' does not provide an export named 'URLPattern'

Unfortunately, this type of issue is one in which you don't know the full extent of the problem. You solve it in one area, just to then reveal the next. You don't know you're done until you are done. So, I can't report in much of any helpful detail right now what the particular issues are.

Offering the browser bundle in the npm package is the safest and quickest way to provide a working solution for browsers. But if really wanting to provide a ESM module setup compatible with browsers, then more work needs to be done to isolate the dependencies in deps.ts according to their compatible environments. Maybe that's all that's really needed to be done.

Answer 6 · 2022-03-02T21:42:17.000Z

More specifically, as a start, you may want to consider splitting up deps.ts into browser-deps.ts, node-deps.ts, and universal-deps.ts. But I'm sure it'll get more complex than that, as you dive deeper into it.

Answer 7 · 2022-03-02T21:46:00.000Z

crypto-driver-chloride.js is a good example of how the dependencies are handled well. It depends on the chloride and buffer packages, but it is only referenced by the node.js entry. As a result, these two dependencies are never encountered by the browser and do not become a problem.

Answer 8 · 2022-03-03T06:56:34.000Z

@basham I won't bore you with the details, but dependencies are already being mapped for different platforms. This is about mapping the right ones to the NPM package.

I have Earthstar bundling in a NPM CRA app as of yesterday. I will start building a Snowpack app and see what I can do.

Answer 9 · 2022-03-03T09:50:42.000Z

I created a new snowpack react app, and imported Earthstar to see what would happen.

But I get build errors for other packages like this:

Duplicate export 'default' (4:16) in snowpack-wrap:/Users/gwil/Projects/snowpack-test/node_modules/fast-deep-equal/index.js

This feels like it's going to a rabbit hole of fixing people's packages to satisfy Snowpack, and I don't think I can devote my time to that right now. I am open to fixes, but I can't do it myself at this time.

Answer 10 · 2022-03-03T14:59:45.000Z

As some additional tests, instead of having Snowpack try to build Earthstar itself, I switched to using CDN module imports. But both tests results in browser errors.

Skypack

Error (same as I described with the local Snowpack build, yesterday):

SyntaxError: The requested module '/_snowpack/pkg/@sgwilym.urlpattern-polyfill.v1.0.0-rc8.js' does not provide an export named 'URLPattern'

ESM.sh

SyntaxError: The requested module '/v66/concurrency-friends@5.2.0/es2021/concurrency-friends.js' does not provide an export named 'Lock'

So, yes, this rabbit hole (as you say) is what I was fearing. It is certainly not worth your limited time trying to resolve all these issues right now. Thanks for exploring it for a bit, though.

The Deno-compiled browser bundle does work. If that is included in the NPM package, that will provide more options to users of Earthstar, with little additional time on your end.

Answer 11 · 2022-03-04T20:17:52.000Z

@basham esm.sh and skypack are definitely going to give you weird results. Importing this way means you're import a module built for ESM, translated to a NPM module, and then translated back to ESM again.

If you're going to use a URL import, why not go back to where this all started?

import * as Earthstar from "https://cdn.earthstar-project.org/js/earthstar.bundle.v7.1.0.js";

This would import the made-for-ESM bundle.

Answer 12 · 2022-03-04T21:41:30.000Z

I do not intend to use a URL import of Earthstar. I just did the Skypack and ESM.sh test as a way to see if those systems somehow could resolve this issue in a way that the other approaches seems to not be doing right now.

I want a local reference to the Earthstar browser bundle so that Snowpack will copy it to the build folder and then Workbox can cache it with a service worker up front. I want all dependencies hosted in the same place, so that there's only one potential place for network errors (where the app is hosted). Also, by installing Earthstar via npm, I can easily know when Earthstar has a new version when I check for new versions of all dependencies (via npm outdated) and manually update them. Taken together, this provides a better offline experience for users, while simplifying the build process and maintenance of the app.

If I would stop using the "basham/earthstar-bundle#v7.1.0" repo, then it complicates other parts of the build process. I would write a Node script to pull down the bundle from the CDN. Then that needs to be saved in a temporary folder. Then I need to make Snowpack aware of it. I'm essentially needing to recreate what npm does for me, for the sake of a single dependency.

Is there a concern with including the bundle in the npm package?