Out of memory
Closed this issue · 9 comments
Fresh install using npm, when trying to launch dtd2mysql with any parameters throws out of memory error.
~$ dtd2mysql
<--- Last few GCs --->
105 ms: Mark-sweep 1.4 (5.5) -> 1.4 (5.5) MB, 3.1 / 0.0 ms [allocation failure] [GC in old space requested].
108 ms: Mark-sweep 1.4 (5.5) -> 1.4 (5.5) MB, 2.6 / 0.0 ms [allocation failure] [GC in old space requested].
110 ms: Mark-sweep 1.4 (5.5) -> 1.3 (5.5) MB, 2.1 / 0.0 ms [allocation failure] [GC in old space requested].
112 ms: Mark-sweep 1.3 (5.5) -> 1.3 (4.5) MB, 2.1 / 0.0 ms [last resort gc].
114 ms: Mark-sweep 1.3 (4.5) -> 1.3 (4.5) MB, 2.1 / 0.0 ms [last resort gc].
Hi, would you be able to tell me the version of node you have and the filename you are trying to import.
Using node v6.11.0 & npm 3.10.10
The same error is encountered with any parameters, e.g.
dtd2mysql
dtd2mysql -v
DATABASE_USERNAME=XXX DATABASE_PASSWORD=XXX DATABASE_NAME=XXX dtd2mysql --fares ./RJFAF607.ZIP
Unfortunately it requires node 8 or above to run. Could you upgrade and try again?
Thank you - I've since upgraded node to v8.9.1
This seems to have solved the overt error, however process still dies with:
Extracting ./RJFAF607.ZIP to /tmp/dtd/fares/
Killed
Running with environment variable NODE_DEBUG=dtd2mysql gives the following error:
RangeError: Array buffer allocation failed
at new ArrayBuffer ()
at typedArrayConstructByLength ()
at new Uint8Array (native)
at new FastBuffer (buffer.js:38:5)
at Function.Buffer.alloc (buffer.js:243:10)
at new Buffer (buffer.js:156:19)
at decompress (/usr/lib/node_modules/dtd2mysql/node_modules/adm-zip/zipEntry.js:56:20)
at Object.getData (/usr/lib/node_modules/dtd2mysql/node_modules/adm-zip/zipEntry.js:241:20)
at /usr/lib/node_modules/dtd2mysql/node_modules/adm-zip/adm-zip.js:377:37
at Array.forEach ()
at Object.extractAllTo (/usr/lib/node_modules/dtd2mysql/node_modules/adm-zip/adm-zip.js:372:26)
at ImportFeedCommand.doImport (/usr/lib/node_modules/dtd2mysql/dist/src/cli/ImportFeedCommand.js:53:30)
at ImportFeedCommand.run (/usr/lib/node_modules/dtd2mysql/dist/src/cli/ImportFeedCommand.js:35:24)
at container.getCommand.then.c (/usr/lib/node_modules/dtd2mysql/dist/src/index.js:7:18)
at
at process._tickCallback (internal/process/next_tick.js:188:7)
How strange, I will bump the memory and release a new version.
I've just released a new version (3.4.1) that will use more memory but that did make me think - how much memory does your system have available?
Upgraded to 3.4.1
System memory is
$ free
total used free shared buffers cached
Mem: 507616 301488 206128 24628 72 42684
-/+ buffers/cache: 258732 248884
Swap: 0 0 0
If I'm reading that correctly you have roughly 200~MB of memory free which I don't think will be enough. Is there any chance that you can free up some memory or try on a different machine?
You're quite right - 200MB available isn't sufficient.
Creating a 1 GB swapfile dealt with the issue :)