Converting a large JSONArray to a string in Java throws OutOfMemory exception
Closed this issue · 1 comments
Bug Report
Problem
My Android app uses this sqlite plugin for data persistence.
The app crashes on starting up with the following stack trace:
java.lang.OutOfMemoryError: Failed to allocate a 150994952 byte allocation with 100663296 free bytes and 141MB until OOM, target footprint 220905960, growth limit 268435456 at java.util.Arrays.copyOf(Arrays.java:3578) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:177) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:753) at java.lang.StringBuilder.append(StringBuilder.java:257) at org.json.JSONStringer.string(JSONStringer.java:354) at org.json.JSONStringer.value(JSONStringer.java:261) at org.json.JSONArray.toString(JSONArray.java:587) at org.apache.cordova.PluginResult.<init>(PluginResult.java:49) at org.apache.cordova.CallbackContext.success(CallbackContext.java:88) at io.sqlc.SQLiteConnectorDatabase.executeSqlBatch(SQLiteConnectorDatabase.java:172) at io.sqlc.SQLitePlugin$DBRunner.run(SQLitePlugin.java:341) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644) at java.lang.Thread.run(Thread.java:1012)
What does actually happen?
The root cause is that it tries to convert a large json object into a string in memory, which leads to out of memory exception.
Even though I haven't found out what piece of data that is so big that can lead to this. fyi I don't store files (image/video/file) in memory
The error traces back to PluginResult.java, line 49.
public PluginResult(Status status, JSONArray message) { this.status = status.ordinal(); this.messageType = MESSAGE_TYPE_JSON; encodedMessage = message.toString(); }
Is there anyway in which we can read the jsonArray in chunks and avoid holding the whole data in memory?
Environment, Platform, Device
Platform: Android
Device: Redmi K70 on Xiaomi HyperOS version 1.0.11.0
Version information
Cordova android version 12.0.0
Is there anyway in which we can read the jsonArray in chunks and avoid holding the whole data in memory?
Kind of, but it will be on the plugin author. Cordova supports APIs that makes use of a callback several times. So the plugin could have an API that retains the callback and invokes it several times. This means the plugin could chunk the response and only send parts of the dataset in reasonably sized chunks back to the webview, until all data is sent over.
Otherwise you'll likely need to page your SQL queries to limit the data being returned, potentially requiring multiple query calls to build the complete dataset.
Even though I haven't found out what piece of data that is so big that can lead to this. fyi I don't store files (image/video/file) in memory
It's probably just the number of records being returned, or the amount of string manipulation occurring. The problem with strings is it requires contiguous memory. Each string manipulation involves allocating a new block of contiguous memory and copying over the old part + the new part to form the new string. Even if you're not actually exhausting RAM, you could still run into OOM simply because the RAM is so fragmented that you cannot allocate the size of contiguous memory to fit the new string.
Even if the native side can fully build the stringified JSON, it needs to send it over to JS which is another copy operation, and inside JS everytime that string variable is passed through to a function it's passed by value for string types, and thus it's copied again.
I'm closing this issue because it isn't a bug with Cordova.