memory leak
Closed this issue · 1 comments
barrybecker4 commented
There were a couple of leaks:
The experience buffers were holding onto memory longer than needed.
The nodes contained a serializer with self-reference.
I could see the memory growing in JVisualVM.
There seems to be a separate serious leak when using 1.0.0-beta4 or 1.0.0-beta6 version of nd4j which I will open a separate issue for.
I lowered memory settings to
"-Xms128M",
"-Xmx256M",
"-Dorg.bytedeco.javacpp.maxbytes=256M",
"-Dorg.bytedeco.javacpp.maxphysicalbytes=1G",
and turned off all experience collecting an training, and it still ran out of memory by episode 9.
Probably the problem is with the building of the nodes.
Maybe go back to 1.0.3 and test.
barrybecker4 commented
fixed in my fork.