JavaScript heap out of memory
KushalKochar opened this issue · 15 comments
Hi Team,
I am facing the javascript heapt out of memory issue every few minutes when I try to run tools/sync.js
This happens when I point to MAIN net (ETHEREUM). Ropsten works fine for me. I am using node v8.11.3 and npm 3.5.2.
Error details:
<--- Last few GCs --->
[4532:0x3cc86a0] 1301138 ms: Mark-sweep 1279.8 (1520.3) -> 1279.6 (1520.8) MB, 1021.1 / 0.0 ms allocation failure GC in old space requested
[4532:0x3cc86a0] 1302072 ms: Mark-sweep 1279.6 (1520.8) -> 1279.6 (1481.3) MB, 933.8 / 0.2 ms last resort GC in old space requested
[4532:0x3cc86a0] 1303001 ms: Mark-sweep 1279.6 (1481.3) -> 1279.6 (1476.3) MB, 928.6 / 0.0 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x381d4ea5879
2: writeTransactionsToDB [/home/project/eth-indexer/tools/sync.js:141] [bytecode=0x279c5aa527c9 offset=85](this=0xe7e24a8c209 ,config=0x19a6076829d1 ,blockData=0x1fe83c2c8d69 ,flush=0x3d30d21822d1 )
3: arguments adaptor frame: 2->3
4: callback [/home/project/eth-indexer/tools/sync.js:84] [bytecode=...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x8c21ec [node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handlev8::internal::Object, bool, int) [node]
6: v8::internal::CodeGenerator::MakeCodeEpilogue(v8::internal::TurboAssembler*, v8::internal::EhFrameWriter*, v8::internal::CompilationInfo*, v8::internal::Handlev8::internal::Object) [node]
7: v8::internal::compiler::CodeGenerator::FinalizeCode() [node]
8: v8::internal::compiler::PipelineImpl::FinalizeCode() [node]
9: v8::internal::compiler::PipelineCompilationJob::FinalizeJobImpl() [node]
10: v8::internal::Compiler::FinalizeCompilationJob(v8::internal::CompilationJob*) [node]
11: v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [node]
12: v8::internal::StackGuard::HandleInterrupts() [node]
13: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [node]
14: 0x16b1e5042fd
Allocation failed - JavaScript heap out of memory
how about to change bulkSize from bulkSize : 100 to bulkSize: 50 or less than 50 at the config.json
Thanks @hackmod . I have restarted the process with suggested changes and will update the result here!
with bulkSize: 50, it didn't worked! I will try with 25 now!
Is it me only, or anyone else faced this issue?
~8GB memory + parity + 120GB SSD with no problem. (ETC, ESN cases)
I am also using the same config:
~8GB memory + parity + 256GB SSD...
I already had the data synced, so I was using ethExplorer for only indexing purpose. Yesterday, I tried with bulkSize of 10 and it ran for whole night (so looks like it is working with this bulksize)!
I will keep this issue open until it indexes the complete data.
@KushalKochar Any success?
Try running with
NODE_OPTIONS="--max-old-space-size=4096"
default for nodejs is about 1.5GB. Judging by your errors you are hitting that limit
[4532:0x3cc86a0] 1301138 ms: Mark-sweep 1279.8 (1520.3) -> 1279.6 (1520.8) MB, 1021.1 / 0.0 ms allocation failure GC in old space requested
[4532:0x3cc86a0] 1302072 ms: Mark-sweep 1279.6 (1520.8) -> 1279.6 (1481.3) MB, 933.8 / 0.2 ms last resort GC in old space requested
[4532:0x3cc86a0] 1303001 ms: Mark-sweep 1279.6 (1481.3) -> 1279.6 (1476.3) MB, 928.6 / 0.0 ms last resort GC in old space requested
I am still facing the same issue with bulk size of 10. Now I will try with NODE_OPTIONS="--max-old-space-size=4096"
I tried NODE_OPTIONS="--max-old-space-size=4096" still getting the same issue. I was out of action for few days but now again back on this project, so will be able to update more regularly on the progress/issue.
Any other ideas/suggestion that I can try implementing?
As mentioned in my original question..... I am trying to build indexer for ETHEREUM (ETH) and ERC tokens.
@KushalKochar do you have the output of the last lines of GCs with this option set?
How much swap is on this device? how much free memory is present after it runs a while?
free -h
Again the same error. Please note that I am using DOCKER to run mongodb and explorer :
`* 1338 transactions successfully recorded.
- block #5908787 inserted.
- block #5908787: 337 transactions recorded.
- block #5908776 inserted.
- block #5908776: 72 transactions recorded.
<--- Last few GCs --->
[8:0x2d0ba30] 17149742 ms: Mark-sweep 1235.0 (1427.8) -> 1235.0 (1388.8) MB, 879.0 / 0.0 ms (average mu = 0.326, current mu = 0.000) last resort GC in old space requ ested
[8:0x2d0ba30] 17150616 ms: Mark-sweep 1235.0 (1388.8) -> 1235.0 (1380.8) MB, 873.2 / 0.0 ms (average mu = 0.195, current mu = 0.000) last resort GC in old space requ ested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x929d379e6c9
0: builtin exit frame: parse(this=0x929d3787949 ,0 x1476841b9541 <Very long string[220249]>,0x929d3787949 )
1: onreadystatechange [0x92b8defa271] [/eth/node_modules/web3/lib/web3/httpprov ider.js:113] [bytecode=0x280042e54321 offset=62](this=0x92b8deb4e11 <XMLHttpRequest map = 0x2d93642e1059>)
2: arguments adaptor frame: 1->0
...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: 0x8b5f80 node::Abort() [node]
2: 0x8b5fcc [node]
3: 0xab730e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xab7528 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xea5152 [node]
6: 0xeb43c6 v8::internal::Heap::AllocateRawWithRetry(int, v8::internal::Allocation Space, v8::internal::AllocationAlignment) [node]
7: 0xe7ca7b v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::Heap::Roo tListIndex, int, v8::internal::Object*, v8::internal::PretenureFlag) [node]
8: 0xe7cb80 v8::internal::Handlev8::internal::FixedArray v8::internal::Factory:: NewFixedArrayWithMapv8::internal::FixedArray(v8::internal::Heap::RootListIndex, i nt, v8::internal::PretenureFlag) [node]
9: 0xfb0e40 v8::internal::HashTable<v8::internal::StringTable, v8::internal::Strin gTableShape>::NewInternal(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [node]
10: 0xfb0e9e v8::internal::HashTable<v8::internal::StringTable, v8::internal::Strin gTableShape>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag, v8::int ernal::MinimumCapacity) [node]
11: 0xfc2a48 v8::internal::HashTable<v8::internal::StringTable, v8::internal::Strin gTableShape>::EnsureCapacity(v8::internal::Handlev8::internal::StringTable, int, v8::internal::PretenureFlag) [node]
12: 0xfc2c0a v8::internal::StringTable::LookupKey(v8::internal::Isolate*, v8::inter nal::StringTableKey*) [node]
13: 0xe7dafe v8::internal::Factory::InternalizeOneByteString(v8::internal::Handle<v 8::internal::SeqOneByteString>, int, int) [node]
14: 0xf68078 v8::internal::JsonParser::ScanJsonString() [node]
15: 0xf6cff8 v8::internal::JsonParser::ParseJsonValue() [node]
16: 0xf6c59b v8::internal::JsonParser::ParseJsonObject() [node]
17: 0xf6d030 v8::internal::JsonParser::ParseJsonValue() [node]
18: 0xf6d4c4 v8::internal::JsonParser::ParseJsonArray() [node]
19: 0xf6d040 v8::internal::JsonParser::ParseJsonValue() [node]
20: 0xf6c59b v8::internal::JsonParser::ParseJsonObject() [node]
21: 0xf6d030 v8::internal::JsonParser::ParseJsonValue() [node]
22: 0xf6c59b v8::internal::JsonParser::ParseJsonObject() [node]
23: 0xf6d030 v8::internal::JsonParser::ParseJsonValue() [node]
24: 0xf6d0a3 v8::internal::JsonParser::ParseJson() [node]
25: 0xb84768 v8::internal::Builtin_JsonParse(int, v8::internal::Object**, v8::inter nal::Isolate*) [node]
26: 0x20f94db8a1dd
./docker_entrypoint.sh: line 36: 8 Aborted (core dumped) node . /tools/sync.js
`
free -h total used free shared buff/cache available Mem: 15G 7.3G 3.3G 8.7M 5.1G 8.0G Swap: 0B 0B 0B
Yeah for some reason node is only running with 1.5GB limits still according to your error. Wont matter how much ram is on the machine, the process is only allowing 1.5GB so when that hits, it crashes
Looks like the 4GB memory also gives the same error:
<--- Last few GCs --->
[9:0x365b1b0] 1701580 ms: Mark-sweep 3659.9 (4042.2) -> 3659.9 (4006.2) MB, 3081.7 / 0.0 ms (average mu = 0.496, current mu = 0.000) last resort GC in old space requested
[9:0x365b1b0] 1704755 ms: Mark-sweep 3659.9 (4006.2) -> 3659.9 (4001.2) MB, 3175.2 / 0.0 ms (average mu = 0.319, current mu = 0.000) last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x3277fea1e681
0: builtin exit frame: parse(this=0x3277fea07931 ,0x0538fc248439 <Very long string[221284]>,0x3277fea07931 )
1: onreadystatechange [0x85721bcc261] [/eth/node_modules/web3/lib/web3/httpprovider.js:~107] [pc=0x10eb9152ebd9](this=0x085721bc5bb9 <XMLHttpRequest map = 0x78c0e1e5fc1>)
2: arguments adaptor frame: 1->0
3: dispatc...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: 0x8d1bc0 node::Abort() [node]
2: 0x8d1c0c [node]
3: 0xb00bae v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xb00de4 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xeee2b2 [node]
6: 0xefdb1f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
7: 0xec6c16 v8::internal::Factory::AllocateRawArray(int, v8::internal::PretenureFlag) [node]
8: 0xec749a v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::Heap::RootListIndex, int, v8::internal::Object*, v8::internal::PretenureFlag) [node]
9: 0xec7540 v8::internal::Handlev8::internal::FixedArray v8::internal::Factory::NewFixedArrayWithMapv8::internal::FixedArray(v8::internal::Heap::RootListIndex, int, v8::internal::PretenureFlag) [node]
10: 0x1001fb0 v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::NewInternal(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [node]
11: 0x100200e v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag, v8::internal::MinimumCapacity) [node]
12: 0x1014edf v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::EnsureCapacity(v8::internal::Handlev8::internal::StringTable, int, v8::internal::PretenureFlag) [node]
13: 0x10150e5 v8::internal::StringTable::LookupKey(v8::internal::Isolate*, v8::internal::StringTableKey*) [node]
14: 0xec843e v8::internal::Factory::InternalizeOneByteString(v8::internal::Handlev8::internal::SeqOneByteString, int, int) [node]
15: 0xfb9b58 v8::internal::JsonParser::ScanJsonString() [node]
16: 0xfbeac8 v8::internal::JsonParser::ParseJsonValue() [node]
17: 0xfbe04c v8::internal::JsonParser::ParseJsonObject() [node]
18: 0xfbeb00 v8::internal::JsonParser::ParseJsonValue() [node]
19: 0xfbef95 v8::internal::JsonParser::ParseJsonArray() [node]
20: 0xfbeb10 v8::internal::JsonParser::ParseJsonValue() [node]
21: 0xfbe04c v8::internal::JsonParser::ParseJsonObject() [node]
22: 0xfbeb00 v8::internal::JsonParser::ParseJsonValue() [node]
23: 0xfbe04c v8::internal::JsonParser::ParseJsonObject() [node]
24: 0xfbeb00 v8::internal::JsonParser::ParseJsonValue() [node]
25: 0xfbeb74 v8::internal::JsonParser::ParseJson() [node]
26: 0xbcf838 v8::internal::Builtin_JsonParse(int, v8::internal::Object**, v8::internal::Isolate*) [node]
27: 0x10eb90edc17d
You will need at least 30 GB of memory allocation for ethereum classic chain
For example,
node --max-old-space-size=4096 tools/sync.js
And try expanding swap memory for your system.