Constant R14 Memory Quota Exceeded with basic Spring Boot application
vicros0716 opened this issue · 6 comments
Hello,
I have a basic Spring Boot application deployed on a hobby tier Heroku instance and it's been throwing R14 Memory Quote Exceeded near constantly. The application is literally a couple REST endpoints connecting to a database.
I've been trying to track this down for a some number of hours across a few days now and don't seem much closer to a solution. I have turned on Enhanced Language Metrics and see the following
My application is started via java -Dserver.port=$PORT $JAVA_OPTS -jar build/libs/*.jar.
Based on https://devcenter.heroku.com/articles/java-memory-issues#configuring-nativememorytracking, I have JAVA_OPTS set to -Xms256m -Xmx256m -Xss512k -XX:NativeMemoryTracking=detail -XX:+UnlockDiagnosticVMOptions -XX:+PrintNMTStatistics in the config vars.
The issue metabase/metabase-deploy#5 and metabase/metabase#3360 make me suspect I'm not the only one having trouble tuning the JVM to run below the 512 MB limit for these Heroku dynos.
Does anyone have more information or suggestions to reduce the memory footprint? It's a little unnerving to see the application consume so much memory for minimal functionality.
Thanks!
What kind of memory profile does the app show locally? Metabase isn't exactly a minimal app, it's a complete working app, which may be difficult to fit into a 512MB limit (in fact, most real world Java apps are hard to fit into a 512 MB limit).
Ah I'm not using metabase. Those github issues just happened to be where I found similar issues reported. The minimal app I have is Spring boot with three REST endpoints that read from/write to a Postgres database.
Locally running java -jar build/libs/*.jar -Xms256m -Xmx256m -Xss512k, the heap memory profile in VisualVM looks like
Running jconsole, I see that the heap space has 378,880 kbytes committed and non-heap space has 101,728 kbytes committed which totals around 480.61 megabytes. It's a little closer than I would like, but should be comfortably under the 512 Heroku provides.
I'm not surprised that the Java app has trouble fitting under the 512 MB limit, but I would definitely expect the -Xmx option to limit the committed heap. Rerunning with -Xmx128m resulted in the same size of committed heap. So I guess the question has turned to why the JVM is not obeying -Xmx.
Move the -Xmx option before the -jar. The way you have it now, it’s being passed as an app argument (the ones you get in main(String[])).
Thanks for pointing that out and all your support!
Putting the -Xmx before the -jar helped me get my local analysis to behave as expected. I was subsequently able to get jconsole to connect remotely with Heroku and experimenting with tuning the JAVA_OPTS and seeing expected behavior on Heroku so I consider this issue resolved and this ticket closed.
Turns out that Heroku adds a ~150mb overhead to the JVM on top of the ~100 mb non-heap committed memory. When I use -Xmx256m, total memory usage on Heroku's metrics hover around 506mb. When I use -Xmx128m, the same metric hovers around 380mb.
In conclusion, the -Xmx option seems to be behaving correctly. Some native memory process on Heroku seems to just eat more memory when running the JVM than I expected and at -Xmx256m was consuming enough memory that regular fluctuations in memory usage were dipping me above the Heroku warning limit.
@vicros0723 it's typical for the JVM to consume more memory on Heroku than locally, but this is not due to any overhead from heroku. Instead, the JVM misunderstands it's limits when it's running in a container (like a Heroku dyno or a Docker container, which are sort of the same thing). Basically, the JVM thinks it has way more memory than it actually has. In Java 9/10/11 this is starting to be improved with the -XX:+UseContainerSupport option, but it really only works for heap (and the JVM has many other categories of memory).
Even though this is a closed issue, I just wanted to add I experienced similar issues. Fighting the Heroku Error R14 (Memory quota exceeded) issue. contained the hint do reduce the default hikari pool size, which had a significant impact on the memory usage of my app. Spring boot 2.3.2.RELEASE, Java 11