regular/unbzip2-stream

JavaScript heap out of memory

Opened this issue · 2 comments

braco commented

Anyone else encounter this?

<--- Last few GCs --->

[35779:0x103001800]   204288 ms: Mark-sweep 1372.5 (1449.5) -> 1372.5 (1449.5) MB, 1296.8 / 0.0 ms  allocation failure GC in old space requested
[35779:0x103001800]   205584 ms: Mark-sweep 1372.5 (1449.5) -> 1372.5 (1418.0) MB, 1296.0 / 0.0 ms  last resort GC in old space requested
[35779:0x103001800]   206876 ms: Mark-sweep 1372.5 (1418.0) -> 1372.5 (1418.0) MB, 1292.4 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2223aa7255e9 <JSObject>
    1: f [/private/var/www/data/node_modules/unbzip2-stream/index.js:~25] [pc=0x3638d944dae9](this=0x2223759866f1 <JSGlobal Object>,b=97)
    2: /* anonymous */ [/private/var/www/data/node_modules/unbzip2-stream/lib/bzip2.js:~163] [pc=0x3638d945853e](this=0x222364102249 <Object map = 0x222302a76661>,bits=0x2223b85b5889 <JSFunction f (sfi = 0x2223d86f2a51)>,stream=0x2223fe782441 <JSFunction f (sf...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/c/.nvm/versions/node/v9.11.1/bin/node]
 2: node::FatalTryCatch::~FatalTryCatch() [/Users/c/.nvm/versions/node/v9.11.1/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/c/.nvm/versions/node/v9.11.1/bin/node]
 4: v8::internal::Factory::NewUninitializedFixedArray(int) [/Users/c/.nvm/versions/node/v9.11.1/bin/node]
 5: v8::internal::(anonymous namespace)::ElementsAccessorBase<v8::internal::(anonymous namespace)::FastPackedSmiElementsAccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)0> >::GrowCapacity(v8::internal::Handle<v8::internal::JSObject>, unsigned int) [/Users/c/.nvm/versions/node/v9.11.1/bin/node]
 6: v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/c/.nvm/versions/node/v9.11.1/bin/node]
 7: 0x3638d92842fd
Abort trap: 6

I have a similar issue. Looks like this module doesn't properly implement backpressure.

Yes, lack of implementing back pressure could very well be the reason. If you write fast to the stream and consume the decompressed data slowly, decompressAndQueue will quickly build up a buffer queue of decompressed data. There probably should be a high watermark that causes the stream to pause. I don't actually use this module, so if one of you feels inclined to implement and test this, I'll certainly be willing to merge a pull request.