Out of memory error during sbt assembly
Closed this issue · 12 comments
As a follow-up to #665 and #667 having been merged, I tried to do sbt -J-Xmx32G assembly on the latest master and keep getting:
...
[info] Including from cache: scala-reflect-2.11.11.jar
[info] Including from cache: bioresources-1.1.30.jar
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
...
Note that I'm using the -J-Xmx32G setting and my computer actually has 20+GB free memory. I also tried to remove my whole .ivy cache but that didn't help either.
I don't think that the size of the ivy cache should be related. The -J-Xmx32G may be telling sbt how much memory to give java when sbt arranges for a java program to run, but it doesn't seem like it would work to increase the amount of memory that sbt itself is getting. Would just -Xmx work? I'm looking at sbt.bat, but you have something else, and it's quite complicated. Is it (or something similar) worth a quick try?
For sbt config, I believe you have to edit this file instead:
https://github.com/clulab/reach/blob/master/.sbtopts
Thanks! Based on your comments I tried sbt -Xmx16g assembly but for that, I get [error] Not a valid command: Xmx16g, I also looked at .sbtopts and changed the first line from -J-Xmx13G to -J-Xmx32G but got the same out of memory error.
I found the sbt file and it's informative reading. You might try export JAVA_OPTS=-Xmx12G and that might be used when starting sbt. Or the switch can be placed in .jvmopts.
Thanks @kwalcock, I tried both of those settings but it didn't help. I also tried to do sbt assembly on a different machine, also with 32G RAM, and got the same out of memory error. Does it work for either of you? This has never been a problem so I suspect that it's due to one of the two PRs I mentioned above, though I'm not sure how.
Well, I can at least confirm I can replicate this issue.
I'll look into it. I'd appreciate help from @kwalcock if he has any cycles available :)
I wonder if this is because now processors-models is 2.2GB...
I'll be looking at this tomorrow when I check on getting Scala 2.11/12 support added.
A couple of updates:
- sbt is indeed using the settings from .sbtopts (I set them incorrectly and it complained).
- I tried setting -Xmx200G and it still ran out of memory. So I suspect it's a different JVM memory zone that needs to be configured, not the main RAM.
There is a bug/feature in older versions of the assembly plugin that make it unable to work with some large files. It was fixed here (sbt/sbt-assembly#312) and released here (https://github.com/sbt/sbt-assembly/releases/tag/v0.14.7). The current version is addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.10") and when I use it, there are no complaints about memory. There is a complaint about something else, a merge conflict. I'm adding code to deal with both problems in a PR. I'm not sure it's the correct solution, but that can be tested afterwards. Assembly completed now after 12 minutes without complaint. Whether the program runs correctly or not I will leave to others for now.
@bgyori: can you please run a quick test?