[all] Phasing out CJS & UMD builds in favor of ESM

[postspectacular]

ESM support has been pretty solid in all major browsers for a while now AND recent versions of NodeJS seemingly cope with these modules very well too (at least in my experience). It's long been a bug bear (and major time waster) for me having to emit 3 different versions (ESM, CJS, UMD) of each package and maintain extra tooling for producing these. I don't know if there're still people out there using CJS or UMD in 2021, but going forward and staying with the times, all packages in this monorepo will only be published as ESM, with these (top off my head) benefits:

• faster builds
  • potentially finally allowing to use TS project references between packages in the repo (maybe 🤷‍♂️ )
• less build infrastructure needed
• smaller packages
• deep/selective imports possible
  • e.g. import { map } from "@thi.ng/transducers/xform/map" vs import { map } from "@thi.ng/transducers"
  • these more precise imports can/will lead to drastically smaller userland file sizes too (some concrete example stats coming soon, early experiments already show file sizes going down to 30-60% of current state (before gzip))...
  • package-level imports will remain possible of course, but each pkg now also defines an export map to expose most of its features

Since this will mean breaking changes for some users (depending on how/where the projects were used so far), this is just a heads up and note to say that intense work is/has already been ongoing (1700+ source files already updated at the time of writing) on the feature/export-maps branch...

The readme files on this branch have already been updated re: installation & import instructions, take a look and let me know please!

As part of this major (structural) update there will also be some further internal restructurings (or even split ups) needed for some of the packages... All in all, a somewhat overdue house cleaning exercise & I hope as such, it's a welcome one for you too!

👍

[postspectacular]

Some promised statistics to illustrate file size savings in the ~100 examples bundled in the repo.

The first table shows a few of the larger examples which have been updated with selective deep imports and their resulting new file sizes ranging between an incredible 5%(!) - 64% compared to previous builds:

example	old	new	ratio
dominant-colors	122.83 KB	61.78 KB	50.3%
fft-synth	171.25 KB	73.20 KB	42.7%
geom-voronoi-mst	132.46 KB	66.58 KB	50.3%
imgui	486.44 KB	111.86 KB	23.0%
parse-playground	397.03 KB	49.25 KB	12.4%
poly-spline	151.96 KB	54.54 KB	35.9%
rdom-dnd	353.82 KB	16.13 KB	4.6%
rdom-lissajous	403.46 KB	32.18 KB	8.0%
rstream-spreadsheet	70.35 KB	44.87 KB	63.8%
shader-ast-raymarch	146.99 KB	83.89 KB	57.1%
shader-ast-workers	94.15 KB	15.14 KB	16.1%
soa-ecs	141.30 KB	75.06 KB	53.1%
text-canvas	103.11 KB	34.61 KB	33.6%
webgl-msdf	169.20 KB	91.98 KB	54.4%
xml-converter	71.79 KB	34.94 KB	48.7%

Some of these more extreme savings are seemingly due to the previous inability of snowpack/webpack to efficiently filter/tree-shake larger packages like the @thi.ng/hiccup-carbon-icons collection (1000+ icons), @thi.ng/vectors (850+ functions) or the @thi.ng/geom geometry package (to name a few). I don't understand why these didn't get optimized properly before, but being able to e.g. directly import only the functions/icons we care about, the problem now disappears entirely. Nice! Most of these import updates are nicely assisted by VSCode and the fact that packages now provide an export map... Only takes a few mins per (lager) example...

The next table shows the impact of the new ESM-only packages without doing any code/import changes to the examples with a range of new file sizes between 37% - 99%:

example	old	new	ratio
adaptive-threshold	101.00 KB	60.36 KB	59.8%
async-effect	31.81 KB	28.59 KB	89.9%
bitmap-font	61.07 KB	40.19 KB	65.8%
canvas-dial	83.37 KB	63.02 KB	75.6%
cellular-automata	22.12 KB	19.26 KB	87.1%
color-themes	121.17 KB	87.34 KB	72.1%
crypto-chart	113.20 KB	64.28 KB	56.8%
devcards	30.75 KB	29.95 KB	97.4%
dominant-colors	122.83 KB	100.50 KB	81.8%
ellipse-proximity	100.57 KB	60.99 KB	60.6%
fft-synth	171.25 KB	120.46 KB	70.3%
geom-convex-hull	114.78 KB	73.51 KB	64.0%
geom-fuzz-basics	140.57 KB	81.25 KB	57.8%
geom-knn	121.43 KB	64.74 KB	53.3%
geom-tessel	152.74 KB	106.38 KB	69.6%
geom-voronoi-mst	132.46 KB	104.33 KB	78.8%
gesture-analysis	158.58 KB	112.29 KB	70.8%
grid-iterators	53.61 KB	35.48 KB	66.2%
hdom-basics	15.11 KB	12.76 KB	84.5%
hdom-benchmark	54.34 KB	33.33 KB	61.3%
hdom-benchmark2	34.44 KB	31.69 KB	92.0%
hdom-canvas-clock	68.83 KB	46.56 KB	67.7%
hdom-canvas-draw	109.76 KB	69.49 KB	63.3%
hdom-canvas-particles	115.51 KB	87.71 KB	75.9%
hdom-canvas-shapes	178.63 KB	151.90 KB	85.0%
hdom-dropdown-fuzzy	40.39 KB	39.12 KB	96.9%
hdom-dropdown	34.08 KB	32.75 KB	96.1%
hdom-dyn-context	28.49 KB	27.70 KB	97.3%
hdom-elm	20.48 KB	18.24 KB	89.0%
hdom-inner-html	15.40 KB	13.05 KB	84.8%
hdom-local-render	56.26 KB	35.19 KB	62.5%
hdom-localstate	16.62 KB	14.37 KB	86.5%
hdom-skip-nested	15.70 KB	13.35 KB	85.0%
hdom-skip	15.43 KB	13.08 KB	84.8%
hdom-theme-adr-0003	17.87 KB	15.62 KB	87.4%
hdom-toggle	17.92 KB	15.02 KB	83.8%
hdom-vscroller	202.50 KB	199.70 KB	98.6%
hiccup-canvas-arcs	137.67 KB	108.42 KB	78.8%
hydrate-basics	31.63 KB	30.31 KB	95.8%
imgui-basics	164.40 KB	106.07 KB	64.5%
imgui	486.44 KB	452.80 KB	93.1%
interceptor-basics	33.26 KB	31.94 KB	96.0%
interceptor-basics2	33.24 KB	32.45 KB	97.6%
iso-plasma	116.03 KB	74.75 KB	64.4%
json-components	17.32 KB	14.51 KB	83.8%
login-form	32.60 KB	32.52 KB	99.8%
mandelbrot	76.97 KB	55.93 KB	72.7%
markdown	93.69 KB	47.67 KB	50.9%
multitouch	109.06 KB	51.36 KB	47.1%
parse-playground	397.03 KB	376.94 KB	94.9%
pixel-basics	46.04 KB	22.01 KB	47.8%
pixel-indexed	79.28 KB	59.76 KB	75.4%
pixel-sorting	114.84 KB	79.80 KB	69.5%
poisson-circles	115.70 KB	87.33 KB	75.5%
poly-spline	151.96 KB	92.70 KB	61.0%
porter-duff	46.34 KB	22.31 KB	48.1%
ramp-synth	73.19 KB	50.94 KB	69.6%
rdom-basics	58.96 KB	35.13 KB	59.6%
rdom-delayed-update	53.57 KB	30.25 KB	56.5%
rdom-dnd	353.82 KB	316.93 KB	89.6%
rdom-lissajous	403.46 KB	346.40 KB	85.9%
rdom-search-docs	68.58 KB	44.85 KB	65.4%
rdom-svg-nodes	96.57 KB	61.09 KB	63.3%
rotating-voronoi	178.61 KB	132.79 KB	74.3%
router-basics	42.66 KB	41.90 KB	98.2%
rstream-dataflow	70.00 KB	50.38 KB	72.0%
rstream-event-loop	55.53 KB	35.33 KB	63.6%
rstream-grid	130.91 KB	78.63 KB	60.1%
rstream-hdom	57.30 KB	36.30 KB	63.3%
rstream-spreadsheet	70.35 KB	54.75 KB	77.8%
scenegraph-image	133.78 KB	NaN bytes	0.0%
scenegraph	117.42 KB	90.38 KB	77.0%
shader-ast-canvas2d	125.25 KB	85.88 KB	68.6%
shader-ast-evo	126.30 KB	80.49 KB	63.7%
shader-ast-noise	145.57 KB	106.43 KB	73.1%
shader-ast-raymarch	146.99 KB	107.86 KB	73.4%
shader-ast-sdf2d	145.52 KB	106.39 KB	73.1%
shader-ast-tunnel	166.55 KB	114.40 KB	68.7%
shader-ast-workers	94.15 KB	35.10 KB	37.3%
shader-graph	141.70 KB	110.36 KB	77.9%
soa-ecs	141.30 KB	97.08 KB	68.7%
spline-tangent	107.47 KB	81.30 KB	75.7%
stratified-grid	115.51 KB	87.13 KB	75.4%
svg-barchart	18.81 KB	16.03 KB	85.2%
svg-particles	20.12 KB	17.31 KB	86.1%
svg-waveform	86.34 KB	48.72 KB	56.4%
talk-slides	76.63 KB	55.58 KB	72.5%
text-canvas-image	50.85 KB	24.57 KB	48.3%
text-canvas	103.11 KB	76.72 KB	74.4%
todo-list	31.31 KB	30.00 KB	95.8%
transducers-hdom	56.52 KB	35.50 KB	62.8%
triple-query	79.99 KB	61.31 KB	76.7%
webgl-cube	137.03 KB	92.55 KB	67.5%
webgl-cubemap	146.12 KB	82.71 KB	56.6%
webgl-grid	137.85 KB	93.26 KB	67.6%
webgl-msdf	169.20 KB	122.85 KB	72.6%
webgl-multipass	136.96 KB	94.81 KB	69.2%
webgl-shadertoy	120.67 KB	73.83 KB	61.2%
webgl-ssao	157.14 KB	139.51 KB	88.8%
wolfram	73.10 KB	52.24 KB	71.5%
xml-converter	71.79 KB	50.75 KB	70.7%

[postspectacular]

All new packages are released and several downstream projects successfully updated. Closing this now.

Ps. Should you run into any issues on your end, please re-open and explain... thx!

[dmnsgn]

Hi @postspectacular,

We (@vorg) are hitting an issue with the use of the "Conditional exports" fields (since v5 in paths and color package at least) when using const paths = require("@thi.ng/paths");:

Module not found: Error: Package path . is not exported from package 
/Users/.../thing-paths-test/node_modules/@thi.ng/paths 
(see exports field in /Users/.../thing-paths-test/node_modules/@thi.ng/paths/package.json)

Quoting Node.js docs, I think using "import" only makes sense when a package exports both cjs and esm:

When using environment branches, always include a "default" condition where possible. Providing a "default" condition ensures that any unknown JS environments are able to use this universal implementation, which helps avoid these JS environments from having to pretend to be existing environments in order to support packages with conditional exports. For this reason, using "node" and "default" condition branches is usually preferable to using "node" and "browser" condition branches.

So this issue is easily resolve by not specifying the condition or adding the default key to the package.json alongside the "import" (@thi.ng/paths example below):

    ".": "./index.js",
    "./api": "./api.js",
    "./delete-in": "./delete-in.js",
    "./get-in": "./get-in.js",
    "./getter": "./getter.js",
    "./mut-in-many": "./mut-in-many.js",
    "./mut-in": "./mut-in.js",
    "./mutator": "./mutator.js",
    "./path": "./path.js",
    "./set-in-many": "./set-in-many.js",
    "./set-in": "./set-in.js",
    "./setter": "./setter.js",
    "./update-in": "./update-in.js",
    "./updater": "./updater.js"

"exports": {
    ".": {
      "import": "./index.js",
      "default": "./index.js"
    },
    "./api": {
      "import": "./api.js",
      "default": "./api.js"
    },
    "./delete-in": {
      "import": "./delete-in.js",
      "default": "./delete-in.js"
    },
// ...
  },

[postspectacular]

Hi @dmnsgn - I don't think I understand what you're asking... For the past 7 months all the packages have ONLY been published as ESM and aren't usable in a CJS environment anymore. So I don't see how an updated export map would help to restore CJS compatibility if the actual syntax in the source files are incompatible with this older format... can you please explain how that would (suppose to) work? Also, may I ask why you're still using CJS?

[dmnsgn]

For the past 7 months all the packages have ONLY been published as ESM and aren't usable in a CJS environment anymore.
if the actual syntax in the source files are incompatible with this older format

The fact that they are published as ESM doesn't mean that they can't be used in a CJS env where loading ESM modules has been backported to the supported Node.js versions. The difference is that you need to dynamically import them.

Simple example with concat-typed-array (which is published as ESM):

package.json

{
  "name": "a-cjs-module",
  "version": "1.0.0",
  "main": "index.cjs",
  "scripts": {
    "test": "node ."
  },
  "dependencies": {
    "concat-typed-array": "^2.1.0"
  }
}

index.cjs

// import concatTypedArray from "concat-typed-array";
// Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
// SyntaxError: Cannot use import statement outside a module

// const concatTypedArray = require("concat-typed-array");
// Error [ERR_REQUIRE_ESM]: require() of ES Module /my-path/a-cjs-module/node_modules/concat-typed-array/index.js from /my-path/a-cjs-module/index.cjs not supported.
// Instead change the require of index.js in /my-path/a-cjs-module/index.cjs to a dynamic import() which is available in all CommonJS modules.

(async () => {
	const concatTypedArray = (await import("concat-typed-array")).default;

	console.log(
		concatTypedArray(Uint8Array, Uint8Array.of(1, 2), Uint8Array.of(3, 4))
	);
        // Uint8Array(4) [ 1, 2, 3, 4 ]
})();

So I don't see how an updated export map would help to restore CJS compatibility
can you please explain how that would (suppose to) work?

My understanding of the error is that because the thing packages provide conditional exports with the "import" key and no "default", bundlers or babel don't know what to do when mixing ESM and CJS (trying to load an esm module inside a cjs one).

The "default" key would allow fallbacks for these not-so straightforward loading cases (from Node.js docs):

"default" - the generic fallback that always matches. Can be a CommonJS or ES module file. This condition should always come last.

Also, may I ask why you're still using CJS?

Avoiding it as much as possible but still indirectly needed inside Nodes as they are written in legacy cjs (although this might change in a near future).

[postspectacular]

Thanks @dmnsgn - have just pushed changes using default instead of import everywhere. Seems to work indeed, didn't know (also didn't want to know, was glad to leave the CJS vs ESM nightmares behind me...) Will do some more tests, then hopefully release later today...

[dmnsgn]

I moved all my packages to ESM last year and looking forward to forget about CJS too.. I have sticked to "exports": "./index.js" though where I export everything public; no conditional exports yet, hoping tree-shaking works correctly.

[postspectacular]

@dmnsgn could you please check the latest pkg versions and let me know if the changes helped your cause? Thank you!

[dmnsgn]

Yes it did! Thanks 🙏

[postspectacular]

Amazing! :)

[stevedevel]

I dont know if this is related but I have a very strange error saying that:

Error [ERR_REQUIRE_ESM]: require() of ES Module C:\Users\xxx\node_modules@thi.ng\grid-iterators\index.js not supported.
Instead change the require of index.js in null to a dynamic import() which is available in all CommonJS modules.

There is no require in my code and as far as I can see and I could not find require in the grid-iterators/index.js. I am using windows, visual studio code and have already deleted caches, reinstalled node_modules, etc.

Do you have any advise?

[postspectacular]

Sorry @stevedevel - I've got no idea what this is supposed to mean. Is your project using CJS? Are you using a bundler? Can you reproduce the issue in CodeSandbox or CodePen etc. or create a temporary dummy repo/gist? Also, which version of Node are you using?

[stevedevel]

Hi Karsten,

Thanks for your reply. I have setuped a simple ts project (on node 16.20):

import { floodFill } from "@thi.ng/grid-iterators";
let structure:number[] = [
1, 0, 0, 0, 0,
1, 0, 0, 0, 1,
1, 1, 0, 0, 0,
0, 0, 0, 0, 0,
1, 1, 0, 0, 0
]
let size = 5;
const region = floodFill((x, y) => structure[x + y * 5] === 1, 1, 0, size, size);

Thats the package.json:

{
"name": "test",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo "Error: no test specified" && exit 1",
"start:dev": "npx nodemon"
},
"author": "",
"license": "ISC",
"dependencies": {
"@thi.ng/grid-iterators": "^4.0.19"
},
"devDependencies": {
"@babel/core": "^7.23.0",
"@types/node": "^20.7.0",
"nodemon": "^3.0.1",
"ts-node": "^10.9.1",
"typescript": "^5.2.2"
}
}

Same error here. There is no require in the code above. I have a large project with all sorts of js/ts code and did not experience this error message. Very strange.

[postspectacular]

I have recreated your project, downloaded & installed node 16.20.2 and can confirm the same error...

However, looking at your package.json, I can also see you'll still need to:

1. add the line "type": "module" to allow your code using ESM imports (unless you want to stay in CommonJS land, and if so, why?)...
2. add some barebones tsconfig.json, something like:

{
    "compilerOptions": {
        "module": "es2020",
        "target": "es2020",
        "esModuleInterop": true,
        "experimentalDecorators": true,
        "moduleResolution": "node",
        "strict": true,
        "strictNullChecks": true,
        "verbatimModuleSyntax": true
    },
   "ts-node": { "esm": true },
    "exclude": ["./**/node_modules"]
}

Once you've got these, you can then run your script via yarn ts-node index.ts or node --loader ts-node/esm index.ts...

(Ps. None of these things have anything to do with this particular project though...)

[stevedevel]

Thank you so much for your answer - greatly appreciated. Especially because this has indeed nothing to do with your code. Now the small test works!

I am using so many 3rd party libs with different JS dialects (CJS, TS, ESM...) within a create-react-app. Thats why I thought there was maybe something related with the library - which is fantastic by the way.

Now my challenge: I cant really modify the toolchain without ejecting and do everything manually :-((

[postspectacular]

Don't envy you & I really have no idea how most JS devs accept & manage to cope with this kind of absolutely insane & unnecessary complexity...

[earthlyreason]

I really have no idea how most JS devs accept & manage to cope with this kind of absolutely insane & unnecessary complexity

That's because you knew better than to accept any dependencies of your own.