split compress and decompress into separate packages
alixaxel opened this issue ยท 6 comments
I'm using iltorb as part of a AWS Lambda provisioning script. Since deployment package size in serverless world is usually quite limited, Brotli is an ideal choice in terms of decompression speed and compression ratio to inflate static binaries and other large files.
iltorb seems to be the best Brotli module in NPM hands-down, but using directly without patching anything, also means the compression that is gained by Brotli is mostly lost due to the iltorb compression dependencies. Publishing a iltorb-decompress package would be super useful for my use case.
I think the challenge here is trying to determine if it is possible to accomplish the following:
- allow us to maintain a single repository for all changes (monorepo)
- maintain a sane build system so that we can provide multiple packages (e.g. iltorb, iltorb-compress, iltorb-decompress)
Here is a quick breakdown of size:
size | description | |
---|---|---|
iltorb | ~14M | contains all dependencies |
iltorb.node | ~900K | our current binary file (includes both enc/dec) |
iltorb/enc | ~772K | binary with only enc |
brotli/enc | ~1.2M | source for brotli/enc (not including common files) |
iltorb/dec | ~216K | binary with only dec |
brotli/dec | ~168K | source for brotli/dec (not including common files) |
With some quick math, we might end up with something like with all of the dependencies installed:
size | description | |
---|---|---|
iltorb | ~14.0M | stays the same |
iltorb-compress | ~13.7M | compression logic only (subtract dec sizes) |
iltorb-decompress | ~12.1M | decompression logic only (subtract enc sizes) |
I removed the node-gyp
dependency since it comes bundled with npm
and yarn
(by using npm
's bundled version). It cuts out ~9M in our package size. While the potential savings of an additional ~0.3M (for a compression-only) and ~0.9M (for a decompression-only) package might be nice, I think it'd unnecessarily complicate the build pipeline to provide/publish two additional packages.
size | description | |
---|---|---|
iltorb | ~5.4M | stays the same |
iltorb-compress | ~5.1M | compression logic only (subtract dec sizes) |
iltorb-decompress | ~3.5M | decompression logic only (subtract enc sizes) |
@oohnoitz I'm using a stripped-down version of iltorb-decompress
on https://github.com/alixaxel/chrome-aws-lambda/tree/master/source/iltorb and it weights less than 250KB.
What accounts for the ~3.5MB you mentioned?
It looks like our dependencies alone would account for ~2.4M, which is pretty much prebuild-install
. This is used to handle the installation logic so that it would grab the proper binaries. The rest would just be the source files to compile our binary when it fails to locate one to download.
In regards to your stripped-down version, it only supports the specific AWS Lambda environment you've compiled it for. It wouldn't work across different versions of Node.js (until we switch to N-API, but that isn't supported by all active LTS versions).
Understood. Well at least iltorb
is now 9MB lighter. ๐ Thanks for looking into this.
Best outcome would be that Brotli support lands in Node.js core (nodejs/node#18964).