Polyglot Crashing
Opened this issue · 8 comments
I've been having an issue where Polyglot just stops, all the nodeservers stay "Connected" but Polyglot is not responding, and all nodeserver heartbeats stop and nothing showed up in the logs. So I ran it in the forground with:
~/polyglot/polyglot-v2-linux-armv7 > out.log 2>&1 &
and then when I came back after I noticed the problem (I was traveling so couldn't check it right away) the terminal showed:
[1]+ Aborted ~/polyglot/polyglot-v2-linux-armv7 > out.log 2>&1
And the last of the log showed:
<--- Last few GCs --->
[29860:0x2c3c748] 321641335 ms: Mark-sweep 227.7 (231.4) -> 227.7 (231.4) MB, 868.9 / 0.0 ms (average mu = 0.877, current mu = 0.003) last resort GC in old space requested
[29860:0x2c3c748] 321642404 ms: Mark-sweep 227.7 (231.4) -> 227.6 (231.4) MB, 1069.7 / 0.0 ms (average mu = 0.790, current mu = 0.000) last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0x14b15cc]
1: StubFrame [pc: 0x140b164]
Security context: 0x2808dbd5 <JSObject>
2: command [0x4d3dc6d9] [/snapshot/polyglot-v2/node_modules/mongodb-core/lib/wireprotocol/2_4_support.js:~149] [pc=0x47a8850c](this=0x4cfc6931 <WireProtocol map = 0x5a12aaf5>,0x70a5f375 <BSON map = 0x4d53ae7d>,0x4cfc9269 <String[20]: polyglot.nodeservers>,0x24b3b929 <Object map = 0x5a11ad85>,0x24b3ba95 <Object map = 0x5a11b325>,0x...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
This was happening with 2.2.3 (although could have been a different error) and still happening with 2.2.4.
1 This problem needs to be figured out
2 We need a way to debug this without having to run Polyglot in the forground
3 The nodeservers need to be started in a way in which they will die with the parent Polyglot also dies
Let me know if you want the entire out.log to debug.
Another user has reported a similar issue.
Not sure if this applies? Seems to say that it's an issue in node 10 and can be fixed by passing memory options?
https://stackoverflow.com/questions/26094420/fatal-error-call-and-retry-last-allocation-failed-process-out-of-memory
export NODE_OPTIONS=--max_old_space_size=4096
Then run polyglot. See if it still crashes.
Thanks, will try that. Also, just noticed node dumped a json file and it says:
"old_space": {
"memorySize": 238096384,
"committedMemory": 237676024,
"capacity": 233911020,
"used": 233911020,
"available": 0
},
Using 4096 is to much for my RPi 3... This is all I get
<--- Last few GCs --->
<--- JS stacktrace --->
#
# Fatal process OOM in insufficient memory to create an Isolate
#
And cutting in half fails with no info
pi@rpi3-2:~/.polyglot/log $ export NODE_OPTIONS=--max_old_space_size=2048
pi@rpi3-2:~/.polyglot/log $ ~/polyglot/polyglot-v2-linux-armv7
11/8/2019, 09:28:14 [polyglot] debug: Created PID file: /home/pi/.polyglot/polyglot.pid
pi@rpi3-2:~/.polyglot/log $ ps -ef | grep poly
pi 11705 1019 0 09:28 pts/0 00:00:00 grep --color=auto poly
modified armv7 build to use node10 vs latest, @jimboca currently testing
Was testing this, but nodeservers were still eventually failing to respond so moved back to 2.2.4 but that is showing the same problem so have no idea what is going on. @Einstein42 and I tried to debug but currently no idea what is happening.
Went back to last build mentioned 2 comments up, and running from command line, will see what happens.