redis-mpool causing the Node process to run OOM when failing to connect to a redis-server
Closed this issue · 2 comments
Hey folks!
We recently ran into an issue when, during a Redis failover that lasted a couple of seconds, all of our SQL API nodes ran OOM and got killed. Here is a stacktrace from one of the nodes:
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x27e609acfb51 <JS Object>
2: create [/home/ubuntu/www/node-sql-api/releases/20180517093803/node_modules/redis-mpool/index.js:~132] [pc=0x2ed135f68a60] (this=0x1beb037c4361 <an Object with map 0x87f26c2f129>,callback=0x23936e1c1f89 <JS Function (SharedFunctionInfo 0x38ac5c14a4d9)>)
3: createResource [/home/ubuntu/www/node-sql-api/releases/20180517093803/node_modules/redis-mpool/node_modules/generic-pool/lib/gener...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/usr/bin/node]
2: 0x112495c [/usr/bin/node]
3: v8::Utils::ReportApiFailure(char const*, char const*) [/usr/bin/node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/bin/node]
5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [/usr/bin/node]
6: v8::internal::TypeFeedbackVector::New(v8::internal::Isolate*, v8::internal::Handle<v8::internal::TypeFeedbackMetadata>) [/usr/bin/node]
7: 0xb61950 [/usr/bin/node]
8: 0xb63cfd [/usr/bin/node]
9: 0xb6c24e [/usr/bin/node]
10: v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag) [/usr/bin/node]
11: v8::internal::Runtime_CompileLazy(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/bin/node]
12: 0x2ed1265092a7
Logs will be written to /home/ubuntu/www/node-sql-api/releases/20180517093803/log/node-sql-api.log
From the stacktrace it seems that, during the Redis outage, the Node process kept trying to allocate memory for new connections (never freeing the old ones) and eventually ran out of memory.
Could you kindly provide more context on this?
Many thanks! 🙏
We use a module to create a pool to handle connections to redis server. This pool is configured with the following options:
// Maximum number of items that can exist at the same time. Default: 1.
// Any further acquire requests will be pushed to the waiting list.
max: 200
// Delay in milliseconds after the idle items in the pool will be destroyed.
// And idle item is that is not acquired yet. Waiting items doesn't count here.
idleTimeoutMillis: 30000
// Cleanup is scheduled in every `factory.reapIntervalMillis` milliseconds.
reapIntervalMillis: 10000
It's really hard to figure out what happened in this case. I guess that during the time of the outage connections were blocked and the pool couldn't determine their status and it couldn't destroy any of them. So new incoming connections were enqueued (it doesn't have a limit) and at some point the process went out of memory. Sadly, we don't have specific metrics of memory usage for sql-api and [system metrics](2018-05-29 07:30:00) doesn't show anything relevant.
It's a corner case, I don't know if you want me to research further but let me know in any case.
I'm going to close this issue, it was a very rare scenario. Reopen if it happens again.