ES modules not supported

I'm getting the following error as soon as the compiled app boots:

node:internal/modules/cjs/loader:930
  throw err;
  ^

Error: Cannot find module '/snapshot/dhjaks/index.js'
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:927:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1776:46)
    at Function.Module._load (node:internal/modules/cjs/loader:772:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1804:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Here is a minimal reproducible example:

package.json

{ "type": "module" }

index.js

import os from 'os'

console.log(os.arch())

Build command:

pkg index.js

You didn't post your complete package.json file. So i think you didn't add your sub folders as scripts. For example if you want to add dhjaks folder as script folder then you need to add this in package file like.
{
"name": "mdm5",
"version": "1.0.1",
"description": "MDM",
"main": "start.js",
"bin": "start.js",
"scripts": {
"start": "node ."
},
"pkg": {
"scripts": [
"dhjaks/*.js"
],
"assets": [],
"targets": [
"node12",
"linux-x64",
"macos-x64",
"win-x64"
]
},
"author": "demo",
"license": "ISC",
"dependencies": {
}
}
assume dhjaks is subfolder under pacakge file parent folder. use build command pkg ./package.json

@Sartaj-Singh I actually did post my complete package.json file ☺️

The dhjaks folder is the folder of my entire package. My entire package only has two files: package.json & index.js. The goal is for it to just print out one line and then exit.

I did it like this to make a minimal test case that shows the problem.

I now tried to use pkg package.json instead:

$ mkdir foobar
$ cd foobar
$ echo '{ "name": "test", "bin": "index.js", "type": "module" }' > package.json
$ echo 'import os from "os"' > index.js
$ echo 'console.log(os.arch())' >> index.js
$ npx pkg package.json
> pkg@5.3.1
> Warning Failed to make bytecode node16-arm64 for file /snapshot/foobar/index.js

$ ./test
node:internal/modules/cjs/loader:930
  throw err;
  ^

Error: Cannot find module '/snapshot/foobar/index.js'
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:927:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1776:46)
    at Function.Module._load (node:internal/modules/cjs/loader:772:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1804:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Check this line:- Warning Failed to make bytecode node16-arm64 for file /snapshot/foobar/index.js
don't use npm or npx, pkg can run independently.
Try to set target in pacakge file like:- "targets": [
"node12",
"linux-x64",
"macos-x64",
"win-x64"
]

don't use npm or npx, pkg can run independently.

npx is just a way to install & run the package without clobbering your global installs. That is not what's causing problems here since I have tried it with a locally installed version of pkg as well.

Try to set target in pacakge file like

Setting the targets doesn't change anything, I've tried different targets and even running on different platforms...

It does however run if I don't use "type": "module", and use require instead of import, so this issue is clearly related to that.

I always create const with require. import statement may be not supported by pkg. I think no need type=module if you compile stand alone executable.

import statement may be not supported by pkg

If it isn't, then this is a feature request

I think no need type=module if you compile stand alone executable.

I need it because I need to import packages which are ESM-only

Hi, could someone clarify clearly on the home page (README.md) whether or not pkg supports ESMs (ES modules) at all? I know it's a free open-source labor-of-love project so I am not demanding anything. It is what it is and it is appreciated as-is. Just would like a clear positioning so we don't need to waste our time trying to package "type": "module" projects, if that's not supported at all. ESMs are not exactly a new invention so a one-liner positioning in the docs would be helpful. If ESM packaging is hopeless with pkg, does anyone know of a workaround (other than rewriting all your code back into CommonJS)? Cheers!

There is the option of using a barebone webpack config to create a single JS file containing all dependencies and not having any external import. Something like this:

const config = {
  mode: "production",
  entry: "./src/main.ts",
  target: "node",
  output: {
    path: resolve(__dirname, "build", "lib"),
    chunkFormat: "commonjs",
  },
};

The output is then usable with pkg.

It should also be possible to update pkg to support ESM; last time I checked I saw two main issues, the babel configuration used (which can be either completely dropped or updated to support module input with a single change), and bytecode generation that failed. Since I already knew of the webpack option I gave up, but fixing bytecode generation with ESM should be doable since node now have full support for it.

For anyone interested I suggest you to firstly use ncc to compile your modules and then use pkg to compile them into executable. There is already an open feature request to include ncc in pkg, maybe with an option

That was what we were doing until a recent update of ncc added compatibility with module-based source. It now produce files that pkg can't use; I could restore the build setup to get the actual error message if needed, but it was something along the line of not handling import statement that were indeed found in the output of ncc.

added compatibility with module-based source

Cannot this be disabled with an option?

Not with an option, sadly. But in the end, ncc basically wraps webpack, hence our solution above. I'm not sure which of the two tools should change, but as it is some features of pkg are simply not used (bundling packages, detecting __dirname, etc.). Still the main feature works perfectly, so it's not so bad.

By double checking the code seems import statements should be supported:

pkg/lib/detector.ts

Line 258 in 59125d1

was = visitorImport(node);

Maybe something isn't working as expected

I tried to look into this but haven't find the root cause, the build process seems to work as the import statement is recognized correctly but then the produced binary isn't working 🤷🏼‍♂️

The exact issue, on a very minimalist project:

have "type":"module" and "bin":"main.js" in package.json
have import fs from "fs"; in main.js
run pkg .

It will output this:

> pkg@5.3.2
> Targets not specified. Assuming:
  node16-linux-x64, node16-macos-x64, node16-win-x64
> Warning Failed to make bytecode node16-x64 for file /snapshot/t/main.js
> Warning Failed to make bytecode node16-x64 for file /snapshot/t/main.js
> Warning Failed to make bytecode node16-x64 for file C:\snapshot\t\main.js

And the binaries are unusable:

node:internal/validators:119                                                                                                                                               
    throw new ERR_INVALID_ARG_TYPE(name, 'string', value);                                                                                                                 
    ^                                                                                                                                                                      
                                                                                                                                                                           
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received null                                                                                
    at new NodeError (node:internal/errors:371:5)                                                                                                                          
    at validateString (node:internal/validators:119:11)                                                                                                                    
    at Object.basename (node:path:1309:5)                                                                                                                                  
    at Error.<anonymous> (node:internal/errors:1462:55)                                                                                                                    
    at getMessage (node:internal/errors:421:12)                                                                                                                            
    at new NodeError (node:internal/errors:348:21)                                                                                                                         
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1128:19)                                                                                            
    at Module.load (node:internal/modules/cjs/loader:981:32)                                                                                                               
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)                                                                                                     
    at Function.runMain (pkg/prelude/bootstrap.js:1804:12) {                                                                                                               
  code: 'ERR_INVALID_ARG_TYPE'                                                                                                                                             
}

Removing "type":"module" and altering the file to use require() produce a working build (but is not acceptable on a large codebase).
Removing "type":"module" while keeping import statement won't work: error while generating bytecode, and the binary output:

(node:288927) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `t-linux --trace-warnings ...` to show where the warning was created)
/snapshot/t/main.js:1
import fs from "fs";
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:354:18)
    at wrapSafe (node:internal/modules/cjs/loader:1031:15)
    at Module._compile (node:internal/modules/cjs/loader:1065:27)
    at Module._compile (pkg/prelude/bootstrap.js:1758:32)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.runMain (pkg/prelude/bootstrap.js:1804:12)
    at node:internal/main/run_main_module:17:47

as expected.

And since ncc was brought up, using ncc on this minimal example and then using its output with pkg yields:

> pkg@5.3.2
> Targets not specified. Assuming:
  node16-linux-x64, node16-macos-x64, node16-win-x64
> Error! import.meta may appear only with 'sourceType: "module"' (5:95)
  /home/cleyfaye/t/dist/index.js

which I traced back to the babel config, and prevent the binary from being build. Quick-fixing this config issue brings us back to the issues described above without using ncc.

That was what we were doing until a recent update of ncc added compatibility with module-based source. It now produce files that pkg can't use; I could restore the build setup to get the actual error message if needed, but it was something along the line of not handling import statement that were indeed found in the output of ncc.

Would you happen to know what specific version of ncc made this change? I am also facing the issue addressed in this issue and am wondering if we could not just downgrade to an ncc version prior to that change and use that?

The change was introduced in with ncc@0.29.0. Since we stopped using it I can't tell if something changed in later releases though.

I haven't dug too deep.. but it looks like pkg wraps whatever program/package is compiled?

@ https://github.com/vercel/pkg/blob/main/prelude/bootstrap.js#L1845

  Module.runMain = function runMain() {
    Module._load(ENTRYPOINT, null, true);
    process._tickCallback();
  };

A minimal test using _load shows:

#~/test$ node testloader.js
node:internal/modules/cjs/loader:1146
      throw err;
      ^

Error [ERR_REQUIRE_ESM]: require() of ES Module ~/test/src/test.js not supported.
Instead change the require of test.js in null to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (~/test/testloader.js:2:8) {
  code: 'ERR_REQUIRE_ESM'
}

Node.js v17.2.0
#~/test$ cat testloader.js
const Module = require('module')
Module._load('./src/test.js',null,true)

So, would we not need to detect here if it's a "type":"module" in package.json or a *.mjs and then import it instead?

Is there a reason it's wrapped this way instead of execing node on the main script? Or is it actually even wrapped like that in the final package? Like I said, I haven't picked too deep on this issue yet but I'd like to help solve it if I can.

@ForbiddenEra if you check linked pr #1323 you will find the reason while esm are not supported yet

@ForbiddenEra if you check linked pr #1323 you will find the reason while esm are not supported yet

I did read all of that, I guess I just (and am still not entirely) don't have full grasp on the process pkg is using. I do plan on possibly pulling the source and digging deeper though.

Now, even if the package was resolved correctly, would we not need a separate runMain for es modules..?

Or, is it the resolver actually generating said runMain function..? or..?

It would be nice if there was a list somewhere of the steps pkg takes exactly, ie:

parse files
lint/compile/byte code
compress

and with which libs/modules any step would involve. pkg seems to work quite differently than I might have guessed, ie, I would've thought that simply it created a self-extracting archive of a node setup and then simply run that node on the script, but there's obviously much more going on here.

I would've thought that simply it created a self-extracting archive of a node setup and then simply run that node on the script, but there's obviously much more going on here.

It's much more complicated then that, caxa does that (but doesn't provide source code protection). For more informations about how it work I have write a developer guide here: https://github.com/vercel/pkg/wiki/Developers

Based on what I have understand the only problem is we are using resolve package to resolve modules but it doesn't support es modules, we should use enhanced-resolve instead. Once that is done es modules should work

I would've thought that simply it created a self-extracting archive of a node setup and then simply run that node on the script, but there's obviously much more going on here.

For more informations about how it work I have write a developer guide here: https://github.com/vercel/pkg/wiki/Developers

Awesome, I must've missed the link to that, I'll check it out.

guys does pkg works if you use dynamic imports?

This is not necessarily a bug.

However, this would be our highest priority feature request.

Any progress here?

Follow updates on #1323. I know @jesec will try to implement it once he has some free time

Well done @jesec 🚀

@jesec Did you opened a PR with that?

No. Still long way to go at this point.

We might have to use vm.Module which is in experimental status as of Node 18.2. I am generally against relying on experimental API in this project. Additionally, bytecode generation and walking of async import is still unresolved at the moment.

keep up the good work!

There is the option of using a barebone webpack config to create a single JS file containing all dependencies and not having any external import. Something like this:
const config = {
  mode: "production",
  entry: "./src/main.ts",
  target: "node",
  output: {
    path: resolve(__dirname, "build", "lib"),
    chunkFormat: "commonjs",
  },
};

Thanks! Hello from the future - I am trying to package a sveltekit application (front end + SSR) in an executable to hand off to a friend so they won't have to install nodejs or anything like that. I encountered an error trying to pkg the output of webpack when it produced multiple files because it creates dynamic imports. So in case anyone is lead to this issue from google, this workaround does work still, but with this change if your webpack is producing multiple files:

const webpack = require("webpack");

const config = {
  mode: "production",
  entry: "./src/main.ts",
  target: "node",
  output: {
    path: resolve(__dirname, "build", "lib"),
    chunkFormat: "commonjs",
  },
  plugins: [
    new webpack.optimize.LimitChunkCountPlugin({
      maxChunks: 1
    })
  ]
};

There are probably consequences for doing this, but it works for now!

@jesec Any update on the status of this?

Is there any update on this?

Is there any update on this?

No ESM support yet but I did manage to get my ESM project to compile by transpiling it to CJS first using esbuild. It works flawlessly though I think if you have any top level async it might break.

Here is the command I use:

esbuild ./src/float.ts --bundle --platform=node --outfile=dist/float.cjs && pkg ./dist/float.cjs --out-path=./build -t latest-linux,latest-mac,latest-win --compress GZip

Is there any update on this?

No ESM support yet but I did manage to get my ESM project to compile by transpiling it to CJS first using esbuild. It works flawlessly though I think if you have any top level async it might break.

Here is the command I use:
esbuild ./src/float.ts --bundle --platform=node --outfile=dist/float.cjs && pkg ./dist/float.cjs --out-path=./build -t latest-linux,latest-mac,latest-win --compress GZip

Works great on my end so far, even with javascript-obfuscator

That's a great workaround @Inrixia. Unfortunately it doesn't work with top level await. If anyone knows a solution, I'd love so much to avoid having to use Webpack or Rollup to bundle a couple simple scripts.

Related issue: evanw/esbuild#253

Another workaround that I'm using here is using rollup npm package like that:

rollup -c && pkg -o program-win -t node16-win-x64 bundle.js

EDIT: I switched to Rust so I don't use Rollup anymore went from 15mb package to 1.5mb. Choose the right tool for the right job

EDIT2: if you still want to see how I used Rollup: MidKnightXI/opgg-ads-remover@70e76ef

I didn't think Rollup could handle top level await. I finally just removed top level await as it was too painful to build.

No ESM support yet but I did manage to get my ESM project to compile by transpiling it to CJS first using esbuild. It works flawlessly though I think if you have any top level async it might break.

Here is the command I use:
esbuild ./src/float.ts --bundle --platform=node --outfile=dist/float.cjs && pkg .

I had an issue with this when using meow which asks for import.meta. you can use the Define esbuild API to make it work.

esbuild bin/cli.js --bundle --platform=node --outfile=dist/cli.cjs --define:import.meta.url=__dirname

#!/usr/bin/env node
// @ts-check

import meow from 'meow'

let url = import.meta.url
// Allow rewriting `import.meta.url` to `__dirname` when bundling with esbuild
if (!url.startsWith('file://')) url = new URL(`file://${import.meta.url}`).toString()

const cli = meow(`
  Usage
    $ your-cli
`,
  {
    // importMeta: import.meta,
    importMeta: { url },
  }
)

Then I could build without any issue.

We might have to use vm.Module which is in experimental status as of Node 18.2. I am generally against relying on experimental API in this project. Additionally, bytecode generation and walking of async import is still unresolved at the moment.

Can --loader help here, once it's stable?

Yeah, transpiling to CJS is not stable because of TLA, actual ES modules will contain top-level await.

That's a great workaround @Inrixia. Unfortunately it doesn't work with top level await. If anyone knows a solution, I'd love so much to avoid having to use Webpack or Rollup to bundle a couple simple scripts.

There is no simple solution except adding bona fide ESM support. Thank you to package maintainers for their very important hard work on this.

for anybody intrested this is how i got ESM working

use node 18 and set file extensions to mjs (index.mjs)
build/convert ESM to CJS using esbuild
esbuild index.mjs --bundle --platform=node --target=node18 --outfile=out.js
set bin in package.json to out.js
use pkg to bundle new CJS file pkg . --no-bytecode --public-packages '*' --public
(optional) create icon.ico and must be less than 135kb
(optional) use resedit-cli to add custom windows data, icon, certifcate etc
resedit --in myapp.exe --out out.exe --company-name "My Company" --file-description "My App Does Stuff" --file-version 1.0.0.0 --icon 1,icon.ico --no-grow --pfx certificate.pfx --password mysecretpassword --product-name "MyApp" --product-version 1.0.0.0 --sign --timestamp "http://timestamp.sectigo.com"

That's basically the same approach as many above, while removing bytecode generation and basically removing all dynamic package loading since it's all bundled in one file. At this point it feels like that bypass most of the features of pkg, aside from the node bootstrap part.

[Off topic] Interesting to see that Node 20's single executable applications suffer from the same limitation.

The single executable application feature currently only supports running a single embedded script using the CommonJS module system.

[Off topic] Interesting to see that Node 20's single executable applications suffer from the same limitation.

The single executable application feature currently only supports running a single embedded script using the CommonJS module system.

I also noticed this. I imagine they'll support ESM within 6 months, it's still experimental.

For now, it should be possible to get a bundle that works with SEAs using ESBuild --bundle option with CJS target and Node module resolution, same as here with pkg. You will just need to refactor any top-level await statements in your program.

Hey,
I'm new to PKG and have the problem with ES modules right now. I wrote my code in ts. So I assume I have to compile it to JS and then wrap it using pkg? I'm using to-level await in my code.
The big problem which I have right now is the conversion or what I have to do exactly. I think I have to compile it to JS to sth like es3, so I don't have those async/await functions in my code anymore. But the tsc gives me a few errors, one of them is I can only use top-level await in newer versions.
Is there any discord server or so to chat, is a bit easier than in here ig.

Thanks in advance!

Hey,
I'm new to PKG and have the problem with ES modules right now. I wrote my code in ts. So I assume I have to compile it to JS and then wrap it using pkg? I'm using to-level await in my code.
The big problem which I have right now is the conversion or what I have to do exactly. I think I have to compile it to JS to sth like es3, so I don't have those async/await functions in my code anymore. But the tsc gives me a few errors, one of them is I can only use top-level await in newer versions.
Is there any discord server or so to chat, is a bit easier than in here ig.

Thanks in advance!

Just use tsc then rollup to convert it to cjs, you should then be able to use pkg.

Hey,
I'm new to PKG and have the problem with ES modules right now. I wrote my code in ts. So I assume I have to compile it to JS and then wrap it using pkg? I'm using to-level await in my code.
The big problem which I have right now is the conversion or what I have to do exactly. I think I have to compile it to JS to sth like es3, so I don't have those async/await functions in my code anymore. But the tsc gives me a few errors, one of them is I can only use top-level await in newer versions.
Is there any discord server or so to chat, is a bit easier than in here ig.
Thanks in advance!

Just use tsc then rollup to convert it to cjs, you should then be able to use pkg.

So the following steps:

tsc index.ts without any TS configuration
rollup index.js --file bundle.js --format cjs to bundle everything together
pkg bundle.js --targets node18-win-x64,node18-linux-arm64 to create the executable?

In short: Doesn't work xD
General improvement: There should be a public chatroom like discord, issues are not made for helping people like me doing things pkg doesn't support out of the box.
When I bundle things together, it only bundles my node_modules with it with a few plugins, then pkg refuses to generate bytecode and so on.
So, my "simple" question: What do I have to do to make pkg work? For me, it looks like I'm very restricted when it comes to writing code that works with pkg because I can't use import x from x or so. When I then try to bundle it or first just compile it to JS which pkg understands, errors about top level await arise…

Well, there are discussions on github now.

About your issue, you can use import statements and as far as I know pretty much anything except top-level awaits as long as your bundler (I use webpack, but other works too) can produce CommonJS. If you need some extra imports that are not bundled directly, you can put them in the virtual fs using pkg config.

Anything that produce a single JS file that works as a CommonJS module will do. Unforutnately, these don't have top-level await.

If you wanna see a project that uses pkg with esm you can look at https://github.com/Inrixia/Floatplane-Downloader

But you cannot use top level await (or have dependencies that use it) as its not possible to transpile that functionality.

So.. I was playing around - not with pkg but with bytecode compiling in general.. I've managed to build a framework that works with CJS, ESM without transpilation, compiling into bytecode w/out any issues.

I haven't dug into pkg enough to figure out where the hold up is, but I just want to point out it's definitely not impossible. I am using experimental module loader for my toy, though I'm not sure if that's a requirement to get it working (sorry, been over a month since I was digging into that)

If you wanna see a project that uses pkg with esm you can look at https://github.com/Inrixia/Floatplane-Downloader

Compiling software that steals from Luke's site? Interesting.. perhaps you should share since they share, but it's not my place to judge or really care besides this snarky comment ;) (especially since I decided to keep my current job instead of going to work for him which was honestly one of the toughest choices of my life)..

Compiling software that steals from Luke's site? Interesting..

@ForbiddenEra
Just to clarify FPD requires a Floatplane account and utilizes the download functionality provided by Floatplane so no stealing here :) though what would I share??

I've worked with AJ on things surrounding it so they are well aware of it's existence too.

Anyway so as not to get too far off topic looking at what you posted about bytecode compilation that's exactly what I expect tbh. I don't see a reason it shouldn't be possible, infact I think there was a working pr submitted for pkg (or one that was wip) but it's been blocked for some time if I'm remembering correctly.

So.. I was playing around - not with pkg but with bytecode compiling in general.. I've managed to build a framework that works with CJS, ESM without transpilation, compiling into bytecode w/out any issues.

I haven't dug into pkg enough to figure out where the hold up is, but I just want to point out it's definitely not impossible. I am using experimental module loader for my toy, though I'm not sure if that's a requirement to get it working (sorry, been over a month since I was digging into that)

Can you share it? I have been working on a bytenode wrapper and it would make things easier to work directly with ESM instead of needing to do a webpack pre-step to convert code to a CommonJS bundle first.

So.. I was playing around - not with pkg but with bytecode compiling in general.. I've managed to build a framework that works with CJS, ESM without transpilation, compiling into bytecode w/out any issues.
I haven't dug into pkg enough to figure out where the hold up is, but I just want to point out it's definitely not impossible. I am using experimental module loader for my toy, though I'm not sure if that's a requirement to get it working (sorry, been over a month since I was digging into that)

Can you share it? I have been working on a bytenode wrapper and it would make things easier to work directly with ESM instead of needing to do a webpack pre-step to convert code to a CommonJS bundle first.

I'll consider it; I can't make any promises, it's not finished and it's been built onto the newest version of my web platform which has always been a commercial product, though I've been considering open sourcing it even if it's at minimum a dual-license kind of thing. And even if I don't open it up, perhaps if I can find some time, perhaps I can poach out a few gists or something on how it works - I was looking at it tonight (as I can only work on this in my spare time currently) and was trying to refresh my memory on things, looks like last time I was working on it I was splitting it up a bit, like having the compiler part into a semi-separate npm module as well as the loaders, perhaps I can even look at open sourcing just those bits once I get it sorted.

The actual compilation part works basically the same as everyone else, eg. bytenode, so I suppose the useful 'magic' is probably in the loaders. I also wasn't quite going for the same goal where it simply outputs a single executable package, though I'm sure that can be made to happen, but at least right now I can do a import testModule from 'testModule.jsc' assert { type: 'jsbin', key: "<key here>" }; where testModule.jsc is a js bytecode binary encrypted using <key here> which was what I was going for in this specific case, though I'm sure others want to just distribute a single compiled file that people can just run, likely with a pre-packaged included node like pkg here or the future SEA will do..

I don't know if it helps but I am using the experimental module loader to allow me to import compiled files anywhere or even standard ts files that get transposed on the fly (which is basically just the example in the nodejs docs for module loaders, heh) and also had to do a bit of magic in the final returned source with vm.SourceTextModule and vm.SyntheticModule and linking with them - I think those are probably the main key actually, I'm not sure using the experimental module loader stuff is needed unless you want to directly import bytecode as I am doing.

Once I've had a chance to dig back in and refresh my memory about this then I'll definitely consider sharing at least a snippet or gist here but with the time since I was working on it and with it being a bit complex and using new/experimental stuff, I don't want to just post something that's not useful or sends someone in the wrong direction or down the wrong rabbit hole! I can't make any promises though as this stuff only gets worked on in my spare time which isn't much lately - but if what I've done can help close this issue then I'll definitely try to share what I can if I find the time.

perhaps I can even look at open sourcing just those bits once I get it sorted.

I think that could be enough :-)

I suppose the useful 'magic' is probably in the loaders.

I think so.

at least right now I can do a import testModule from 'testModule.jsc' assert { type: 'jsbin', key: "<key here>" }; where testModule.jsc is a js bytecode binary encrypted using <key here> which was what I was going for in this specific case

This looks REALLY interesting, and I have been trying to get a similar functionality for Mafalda SFU. I have yet not get into the encription / signing part, in part because I was more interested on a licenses model with expiration date, both for libraries and final executables, but definitely it's something I was thinking about.

I'm sure others want to just distribute a single compiled file that people can just run, likely with a pre-packaged included node like pkg here or the future SEA will do

Yes, but also binary libraries protected with a license or a key can be useful too.

Once I've had a chance to dig back in and refresh my memory about this then I'll definitely consider sharing at least a snippet or gist here but with the time since I was working on it and with it being a bit complex and using new/experimental stuff, I don't want to just post something that's not useful or sends someone in the wrong direction or down the wrong rabbit hole! I can't make any promises though as this stuff only gets worked on in my spare time which isn't much lately - but if what I've done can help close this issue then I'll definitely try to share what I can if I find the time.

Definitely it's something I would be interested about :-) I don't have published my tool as open source too, in part to don't provide tips to somebody willing to reverse engineer my code, but have it totally isolated from my main code and would be easy to integrate something like this.

I think that could be enough :-)

Will see what I can do.. Going on vacation here soon, if I find myself bored one night in a hotel room with my laptop maybe I'll look into it.

Admittedly though, I wonder if I'd almost prefer to actually clone the repo and dive into the problem and see about submitting a PR. I don't know if/when I could dedicate the time but if I provide the solution as a PR then I'm sure I can probably get listed as a contributor, whereas if I provide the solution in an issue then I likely wouldn't be considered a contributor. Not trying to be selfish here but it would suck to provide a solution and have someone else copy/paste it in and get the credit. Hopefully that doesn't seem unreasonable or selfish at all.

I think so.

Definitely part of it, but I was peeking after/while writing my reply, some of it is definitely also vm.*Module stuff for loading compiled modules nicely but also the whole path finding/module resolution thing is definitely the loaders part. I definitely had to have some fun and squeeze some secret sauce with the vm module stuff to be able to properly import compiled modules.

This looks REALLY interesting, and I have been trying to get a similar functionality for Mafalda SFU. I have yet not get into the encription / signing part, in part because I was more interested on a licenses model with expiration date, both for libraries and final executables, but definitely it's something I was thinking about.

Yeah, one of my reasons for implementing was the desire to be able to distribute packages that can be partly or fully bytecode as well as compressed and encrypted with various encryption methods for licensing purposes. I hadn't thought about expiration in the way of self expiring licenses at all but I was thinking about a license server kind of thing. Any type of protection I would deem realistic in the real world would be a rather difficult discussion with JS, one can reverse bytecode just like one can disassemble a typical executable ABI program and in some respects it's potentially even easier. And even without reversing, if you can run the JS, you can debug it pretty thoroughly regardless.

In a lot of cases though I assume that even some level of protection might be enough, at least if your target clientel is moreso corporate or business clients and not the general public as often that target audience won't want to risk non-compliance but could still happen if all a developer has to do is comment out a few lines of license-related code.

When it comes to the general public, again this is JS. I've put a lot of thought into this and the best solution I could come up with that would have any real level of protection would involve a license server. Without that, it can be tough. If using a license key, you have to already deal with all the traditional issues (sharing keys, whether keys expire, whether keys are tied to any system or activated in any way, etc) but also even using the experimental loader stuff requires at least one loader 'layer' to be raw JS in a way that vanilla node can run it, even if you offload the more fun stuff to a second loader that maybe is decrypted or something by the first, one way around this I can think of would be including some sort of actual standalone binary that handles part of the decryption step. At very least your first level loader needs to be executable by native node and without other code handling the bytecode part (which is what you want to use the loader for anyway) that code has to be interpreted by node, thus plaintext JS and you'd at very least be giving away how you load bytecode compiled files even if decryption is done by a binary or following loader layer. At the very least I think you'd need a small loader to load the bytecode into node appropriately of the real loader that might handle the more fun stuff like decryption, etc...but again, JS is JS, if you're serious about protection then you might also want to consider if any V8 options might modify the bytecode from 'standard' if any do to make it more difficult to reverse and looking at ways to prevent users importing the code from being able to run the debugger across it, though if you prevent debugging across the whole importing app you might get some annoyed devs. Then again, it's also JS and I don't think a lot (definitely not all!) of JS developers even know what a debugger is ;)

Some other interesting things you can do though is code signing (which could of course work in conjunction with encryption/keying) where you don't run the code if it doesn't match it's hash/checksum and/or use that hash/checksum as part of your decryption key, again you'd have to obfuscate your decryption somehow.

You could encrypt and sign with a private key and distribute a public key for use, this would prevent easy modification of the code but doesn't prevent anyone obtaining the public key from running it in general, although this could be useful as a security feature maybe? I mean, we already are using SHA hashes for JS on the browser side especially to verify code delivered by CDNs, I feel like this could be pretty easily implemented on the server/node side as well with this method, after all, it's not like we haven't seen attacks on misspelled/mistyped or abandoned npm packages in the past, though you'd probably want a better way of distributing said hash/checksum than just tossing it in your package.json if it were desired to protect against that, but definitely something you could do and I feel like the import assertions-style syntax is ripe for these types of usages, hence why I used it for providing decryption/license keys in my system.

Of course, you can also use the loaders to transpile source on the fly in a way and/or pre-compile it, again this example is in the docs for the loaders as it is but being able to use JS, TS, JSX, TSX in a project without having to think about it or ever transpile anything myself with the option of having the result compiled into bytecode immediately is nice to have and I've also used the import assertions-style syntax to assert the filetype is what's expected regardless of it's extension, though it can be detected by extension as well of course but I also feel like .jsc, '.jsbin, .tsc` etc aren't particularly standard/well-known, so why not allow whatever and use that syntax to assert/specify what is what. You could in theory even use it to specify additional/specific options for transpiling a certain typescript import.

Yes, but also binary libraries protected with a license or a key can be useful too.

Indeed; I wish there were an easier solution and again it's something I've put a bit of thought into and worked on a bit; it can be difficult to protect against things and envision the perceived attack surface when you're the one who developed the protections and know how to side-step them easily, and again the nature of JS doesn't particularly help us here but I'm open to ideas and discussion on how we can try and protect our code where needed, that's partly why as well I was trying to make it in a way where you can just have a single file or module compiled/encrypted, sometimes the whole project doesn't need it but that can still be an option as well.

Definitely it's something I would be interested about :-) I don't have published my tool as open source too, in part to don't provide tips to somebody willing to reverse engineer my code, but have it totally isolated from my main code and would be easy to integrate something like this.

At the very least, I was considering releasing it publicly for use even if I don't release the source so that others can compile their code, encrypt/license it and have a loader to use it in projects or allow other projects to use it. I'm not sure if or when that might happen and I'm not particularly comfortable mentioning it in this issue thread anyway - this of course isn't the place to promote my own work.

Aside/back to original topic in the light of trying to help here:

I'd have to review the thread again but IIRC and if I'm understanding right, the biggest issue was ESM module resolution issues, right? Using the experimental loader stuff can definitely help with that, but that's not the only roadblock I ran into as you can't load a compiled ESM module the same way you load a CJS/standard script, you have to use the vm module stuff as I mentioned above.

One can use the loaders and benefit from nodes resolution though, if that's the primary issue then I'd this guidance might push things forward - although, currently the loader stuff is experimental and I'm not sure if the project maintainers want to go there, however, I don't know if there'd be an alternative without figuring out resolution on your own and trying to ensure it's on par with node's and I'm not sure about others but myself personally would have/would be willing to accept using an experimental feature if it enabled ESM here.

I see that both Deno and Bun can create executables with ESM support.
https://deno.land/manual@v1.36.0/tools/compiler
https://bun.sh/docs/bundler/executables

I haven't tested this yet myself but curious if anyone else has?

It should be noted that both Deno and Bun aren't doing the same thing as pkg. They're completely different runtimes and do things differently from the node runtime. It's not a simple case of "seeing what deno and bun do under the hood and lifting it". It's apples and oranges. Not saying, @Jordan-Eckowitz, that's your implication but just figured I'd say now, so others don't get the wrong idea.

I've tried both Deno and Bun for some of my use-cases. IMHO they're good for smaller projects, but if you have a larger projects with predefined outcomes with the expectation they're drop-in replacements for nodejs/typescript you're in for a bad time.

It should be noted that both Deno and Bun aren't doing the same thing as pkg. They're completely different runtimes and do things differently from the node runtime. It's not a simple case of "seeing what deno and bun do under the hood and lifting it". It's apples and oranges. Not saying, @Jordan-Eckowitz, that's your implication but just figured I'd say now, so others don't get the wrong idea.

I've tried both Deno and Bun for some of my use-cases. IMHO they're good for smaller projects, but if you have a larger projects with predefined outcomes with the expectation they're drop-in replacements for nodejs/typescript you're in for a bad time.

Agreed; I was excited to see both when I discovered them but it was pretty quickly obvious that neither were quite ready for use in any projects that I'm involved with yet and even new ones would, as you said, likely have to be something smaller, not to knock their hard work - they should definitely continue, but the community and ecosystem need to be on board and keep up as well, it was many, many years before I was willing to use node even vs. a standard web server and CGI and not all the concerns I had about switching to node have been resolved or were even resolvable.

Although (and I'm sure it's been stated) node itself has plans for some sort of SEA-ability; whether that will be equivalent to Deno/bun's attempts in this space or competes/replaces things like PKG here I suppose is still to be seen.

As an aside, I've not heard any comments back on whether the maintainers or community would be for or against using/requiring/allowing the use of a loader (as they're still marked experimental) to accomplish the ability; if everyone's against using anything experimental for this, I can understand but then there's not much sense in sharing my solution unless/until loaders are no longer marked experimental?

@ForbiddenEra I've been exploring solutions for a new greenfield project, and the main challenges I've been facing revolves around desktop deployment + licensing. In terms of deployability, for my purpose, Node/Typescript, on the face of it seems like the obvious choice for the projct because it'll run on pretty much anything under the sun with pretty decent platform parity.

One of my desires for the project was to build it ESM First, but honestly, even getting typescript to work properly in ESM mode with third party dependencies was a challenge. Especially those that have taken the route of writing their library CommonJS First, adding TS types and ESM compatibility aliases at a later date. I had serious difficulty importing AJV, for example, to the point I was considering literally rewriting the entire damn thing in typescript from scratch.

ESM loading on the whole seems to simply have too many quirks for me to even consider using it for a new project. On paper, I'm convinced it's the standard we as the community should be following when building libraries, but it just looks like adoption isn't quite there yet to build an end product as ESM. If I was developing the entire thing in-house, zero dependency style, then sure, I'd probably risk it.

In the grand scheme of things, whether a project is deployed as CommonJS or ESM is largely a technical niggle at best. It doesn't affect the broader execution of the developed software, CommonJS isn't deprecated, it's not going anywhere any time soon. It's adequate. Debate me, but I think that's where my gut feeling is for now. Happy to discuss with anyone who disagrees.

That said. Bringing this conversation back to the scope of pkg ... The main purpose of pkg (and kin) is to create a single, deployable executable. I feel that between the methods used by pkg, caxa, nexe, electron-builder this aspect is a reasonably solved issue. The bit that isn't largely solved for JS/Node is the topic of licensing, DRM, source code protection. I had a look into it for the purposes of my own project. Bytecode compilation is pretty much the best we have right now, which can be decompiled with ghidra.

The problem with using loaders for encryption in the manner @ForbiddenEra describes:

import testModule from 'testModule.jsc' assert { type: 'jsbin', key: "<key here>" };
where testModule.jsc is a js bytecode binary encrypted using <key here>

You will end up with a full decrypted copy of whatever you load in memory, which is then passed to the interpreter. This memory could then be read verbatim, saved to a file, then decompiled as usual. In fact, it might be possible (I'm not 100% sure on the logistics), but it might simply be possible to read the executing code from node's v8 code cache. Which would be available regardless of any encryption at any relevant time accessible by loaders. You might be able to mitigate that attack surface by running node in jitless mode, but I'm not sure what effect that would have on pkg. My understanding is that pkg takes a snapshot of the v8 cache and re-seeds it at runtime? So, if the node exec is running jitless, it doesn't pre-allocate the executable memory, so, possibly it'd just barf? Someone more intimate with v8 & pkg chip in if possible?

I'm not saying that custom loaders couldn't be part of the solution, in fact, I think they're the best we'd get without direct access to the AST.

In terms of my thoughts about what pkg's role in this would be, I feel that it'd be somewhat out of scope for the project. However, if pkg could support passing through the experimental loader flag, that might be in-scope.

Alternatively, my next thought would be to create a native plugin that reads encrypted snapshots, and runs them in an isolated worker thread, basically doing what I understand pkg to do. Same caveats as above, wouldn't need custom loader though. Food for thought.

Perhaps this off-topic talk re: licensing could be moved to a discussion? It's interesting, but not what this ticket's for.

ESM loading on the whole seems to simply have too many quirks for me to even consider using it for a new project. On paper, I'm convinced it's the standard we as the community should be following when building libraries, but it just looks like adoption isn't quite there yet to build an end product as ESM. If I was developing the entire thing in-house, zero dependency style, then sure, I'd probably risk it.

I've been trying to use it as much as possible for new stuff without too many issues. I had many more issues with loading CJS stuff in an ESM project.

The problem with using loaders for encryption in the manner @ForbiddenEra describes:

import testModule from 'testModule.jsc' assert { type: 'jsbin', key: "<key here>" };
where testModule.jsc is a js bytecode binary encrypted using <key here>

You will end up with a full decrypted copy of whatever you load in memory, which is then passed to the interpreter. This memory could then be read verbatim, saved to a file, then decompiled as usual.

This is mostly true and something I've considered and thought about how one could work around it but regardless, in the end, you'll be at best feeding bytecode into node, which as you say can be decompiled without too much difficulty.

Fact is, JS is an interpreted language which only makes these things much more difficult. Even a different interpreter, say Bun as a presently-relavent example, even if some sort of protection was a core feature, it still uses JavaScriptCore just like Node uses V8. Beyond that, what do we do? Compile an AST to ASM or WASM?

Not that compiled languages can't be decompiled as well, but when running natively you can use security features of the OS and do things like self-modifying code but the software licensing security problem is far from solved in any domain, the closest I can think of is always online activation perhaps with some additional tricks to check code isn't modified and prevent packet capture/replay/simulate-style attacks like running a hacked local license server.

In terms of my thoughts about what pkg's role in this would be, I feel that it'd be somewhat out of scope for the project. However, if pkg could support passing through the experimental loader flag, that might be in-scope.

It could at least solve the ESM issue with an implementation like I've created.

Alternatively, my next thought would be to create a native plugin that reads encrypted snapshots, and runs them in an isolated worker thread, basically doing what I understand pkg to do. Same caveats as above, wouldn't need custom loader though. Food for thought.

I had a thought along those lines to try and convert/compile JS to WASM in some way. But that's a really deep rabbit hole for the time I have available.

Perhaps this off-topic talk re: licensing could be moved to a discussion? It's interesting, but not what this ticket's for.

I don't disagree; I'm just not sure where. I've tried to mostly stay on-topic while answering questions and offering a bit of extra context regarding what I've put together, though really it was mentioned because it was able to do what pkg does regarding compiling/saving/reloading bytecode on Node but with ES modules.

tsc index.ts without any TS configuration

rollup index.js --file bundle.js --format cjs to bundle everything together

pkg bundle.js --targets node18-win-x64,node18-linux-arm64 to create the executable?

This is working for me. The only thing I needed to add was a jq hack to temporarily remove "type":"module" from package.json so that node would not complain.

cat package.json | jq 'del(.type)' > /tmp/package.json && mv /tmp/package.json package.json # Workaround: remove "type": "module" so node does not complain about require in cjs
rollup dist/bin/tunnelmole.js --file tunnelmole-bundle.js --format cjs
git checkout package.json # Remove workaround, set package type back to module
pkg tunnelmole-bundle.js --targets node18-linux-x64 --output tmole-linux

tsc index.ts without any TS configuration

rollup index.js --file bundle.js --format cjs to bundle everything together

pkg bundle.js --targets node18-win-x64,node18-linux-arm64 to create the executable?

This would be better to avoid js files interrupt your working space

tsc index.ts --outDir dist
rollup dist/index.js --file dist/bundle.js --format cjs",
pkg dist/bundle.js --out-path dist

"scripts": {
  "build": "npm-run-all build:*",
  "build:1": "tsc index.ts --outDir dist",
  "build:2": "rollup dist/index.js --file dist/bundle.js --format cjs",
  "build:3": "pkg dist/bundle.js --out-path dist"
}