lz4 compression while building `totals` is resource intensive
ChALkeR opened this issue · 3 comments
ChALkeR commented
This needs investigation, but something looks wrong with lz4-compressed streams while gzemnid extract totals
is running — the app consumes all the available memory and crashes.
Disabling the compression in config.local.json
and compressing those files manually with command-line stand-alone lz4
works and can be used as a work-around.
ChALkeR commented
Also, there seems to be problems with Node.js 10.x compat for lz4
module: pierrec/node-lz4#62.
I observed those errors being hit from Gzemnid.
ChALkeR commented
Seems to be a problem on node-lz4
side?
This does not cause high memory load:
const outstream = fs.createWriteStream(`${file}.gz`);
const passThrough = new stream.PassThrough();
const gzip = zlib.createGzip({ level: 1 });
passThrough.pipe(gzip).pipe(outstream);
return passThrough;
This causes high memory load:
const outstream = fs.createWriteStream(`${file}.lz4`);
const encoder = lz4.createEncoderStream({ highCompression: false });
encoder.pipe(outstream);
return encoder;
lz4 is significantly faster than gzip though.
ChALkeR commented
Blocked against nodejs/node#21967.