Possibility to reduce packages size by 4x

[pi0]

Hi! Just done a quick test on windows builds. Bundle size can be reduced from 24,964 KB to 6,646 KB (3.75x smaller) with doing simple xz compression. This may even help more when having more npm dependencies. Adding this option may be super useful for small and embedded CLI/Webservers :) I've just opened this discussion before going further to know others idea on this.

UPDATE: To be clear, this topic and what i am talking about is SFX packages not js minification & optimizing.

[pascalandy]

@pi0 Do you see any potential impact your NodeApp's performance?

I'm considering this for my Ghost installation. Cheers!

[roccomuso]

@pi0 How did you do that compression? could you provide an example?

[jgoux]

Hi,

as we're talking about bundle size, does pkg apply some kind of optimization / tree shaking / minification internally?

If I apply something like prepack + uglify before packaging, will my final packaged app be lighter?

[anmonteiro]

running strip on the generated binary should also help on UNIX platforms

[igorklopov]

The idea behind pkg is to introduce minimal changes over original nodejs. Btw, if you run strip you can lose the ability to use native addons (.node files).

[pi0]

Hi. Sorry answering late :)

@pascalandy About performance impact: I think if we do extraction on startup, it may only affect startup time and not run-time itself.

@roccomuso My first test was just using tar! But it would be easy adding self-extract bits on binary's head.(exe inside exe)

@anmonteiro About js minification, pkg is absolutely not webpack! And i don't think this would help a lot, main size suffer is from Node.js binary itself. And we could just make a blind compression on the whole things. (Also @igorklopov is correct about .node files)

[pi0]

Some interesting projects we could get idea/ use:

• https://github.com/megastep/makeself
• http://www.peazip.org/self-extracting-archives.html

[lenovouser]

@anmonteiro strip doesn't really work. It reduces the binary size but the app doesn't run anymore. You get the interactive node console 😆

That is very weird. Why is this happening?

[igorklopov]

strip strips payload that contains your project packed. Without that payload the binary behaves like original node.js binary.

[anmonteiro]

Oh hrm, sorry for the bad suggestion. I use strip with Nexe and it works, apparently it doesn't with pkg's approach. Sorry for the noise

[astefanutti]

Referencing #121 for UPX issue (https://github.com/upx/upx).

[roccomuso]

Btw I wrote a node.js wrapper for UPX, https://github.com/roccomuso/upx

[beeing]

What about the idea of using libsquash similar to https://github.com/pmq20/node-packer ?

upx may cause false positives on some antivirus detection

[Clement-TS]

It seems that the nexe owner gets a clue on this: nexe/nexe#366 (comment)

He uses UPX over node.js itself (~/.nexe/linux-x64-8.9.4), and I can say it works.
However this do not work on the pkg equivalent: ~/.pkg-cache/v2.5/fetched-v8.9.0-linux-x64:

$ pkg -t node8-x64 app.js -o dist/app
> pkg@4.3.0
assert.js:42
  throw new errors.AssertionError({
  ^

AssertionError [ERR_ASSERTION]: Placeholder for not found
    at injectPlaceholder (~/project/node_modules/pkg/lib-es5/producer.js:217:38)
    at injectPlaceholders (~/project/node_modules/pkg/lib-es5/producer.js:236:3)
    at ~/project/node_modules/pkg/lib-es5/producer.js:158:9
    at FSReqWrap.oncomplete (fs.js:135:15)

Any idea?

[roccomuso]

@Clement-TS I got the same error trying to compress the node binary before using pkg. Let's wait @igorklopov

[marcotuna]

Tried today with upx 3.95 on Linux and the problem still persists.
upx -oNodeJSProjectOutput --overlay=copy --strip-relocs=0 --compress-exports=0 --compress-icons=0 --compress-resources=0 --no-align NodeJSProject

[ofarukcaki]

My windows .exe build size reduced 7.xx MB from 21.xx MB with upx, thanks for mentioning it 🌟

[pascalandy]

EDIT: Here is an example of how I use upx to compress Nodejs in my Dockerfile for Ghost: https://github.com/firepress-org/ghostfire/blob/3.0.3-alpine/Dockerfile#L49

[Zireael]

My windows .exe build size reduced 7.xx MB from 21.xx MB with upx, thanks for mentioning it 🌟

Can you say what parameters you used to pack Windows .exe? I tried (UPX 3.95 Win10x64)

upx -o"V:\packed.exe" --overlay=copy --strip-relocs=0 --compress-exports=0 --compress-icons=0 --compress-resources=0 --no-align "V:\app.exe"

but the executable is failing to start.
Thanks in advance!

[ofarukcaki]

My windows .exe build size reduced 7.xx MB from 21.xx MB with upx, thanks for mentioning it star2

Can you say what parameters you used to pack Windows .exe? I tried (UPX 3.95 Win10x64)

upx -o"V:\packed.exe" --overlay=copy --strip-relocs=0 --compress-exports=0 --compress-icons=0 --compress-resources=0 --no-align "V:\app.exe"

but the executable is failing to start.
Thanks in advance!

I didn't use any special flags. Just dragged my .exe and dropped over upx. Thats what I did

[Zireael]

I didn't use any special flags. Just dragged my .exe and dropped over upx. Thats what I did

Thanks, but just Drag&Drop didn't work for me. I'll try packaging with Nexe later, I've read somewhere that they were natively optimised for upx (and support customising .exe file icon out of the box)

[MarcMagnin]

Hey, any update on the matter? UPX + pkg doesn't seems to work so far and that would be really neat if it was.

[Wallacy]

FWIW:

Just tested here using gzexe (Linux) and works good. From 134Mb to 43MB, and i good only 1s performance penalty on load (did not test other runtime operations).

[ArkadiuszMichalski]

Any progress here? When I compress node.exe then it works, but whe I use pkg I always get Pkg: Error reading from file., even for simple console.log().

[Hypfer]

Regarding UPX:
https://sourceforge.net/p/upx/discussion/6806/thread/79e2a6b8/
That also explains why it works fine on windows, since PE executables and their overlays can be and are handled by the upx code.

As it is now, the readPrelude function patched into the nodejs binary will open /proc/self/exe and try reading the overlay from the offset replaced into the binary at packaging time. However since the binary found at /proc/self/exe is the compressed version, all those offsets are of course just wrong and even if they weren't they would still read compressed data.

To get pkg to work with upx, one approach could be to update the readPrelude function so that it uses relative offsets from the end of the file and extend the pkg process so that it first bakes in the commandline options, then packs that binary and finally appends the rest of the code.

That would still result in the js code itself being uncompressed, however, it would at least be able to shrink the nodejs binary by a fair amount.

Another approach could be to change the way the js payload is stored in the final binary so that it is accessible by the code in the readPrelude function directly instead of it calling fs.readSync on /proc/self/exe. That way, it might even be possible to keep the pkg build process as it is.
Since we do seem to be able to template the offsets into that function, why not template in the whole code?

Edit: because it will segfault if the length of that string changes. :)

[Hypfer]

I've just checked a built binary in a hex editor and discovered that

FILESIZE - PRELUDE_SIZE = PRELUDE_POSITION

as well as

FILESIZE - PRELUDE_SIZE - PAYLOAD_SIZE = PAYLOAD_POSITION

Is there any reason I'm missing for why these two positions are stored at all?