Memory Leak

[ourcookiemonster]

Greetings Dibyendu,

Thank you for this very useful wrapper to NLopt. I am seeing that there may be a memory leak. When I run .optimize() on a problem with about 1000 variables the program rapidly grows in memory and eventually runs out (around -Xmx16000M)

If this project is still of interest and you are able to take a look I can upload a sample class that demonstrates the issue.

Kind regards,
Richard

[dibyendumajumdar]

Sure please post a test case

[ourcookiemonster]

Thanks Dibyendu,

Here is a source file that replicates the issue.
NLoptOOM.txt

The custom execute methods simply perform calculations and do not allocate any objects, Memory use continues to grow and grow while the optimizer runs and also does not get recaptured after the optimization finishes. Gradient and non-gradient both impacted.

If there is an issue I feel like it ought to be in this function:

double nlopt_minfunc(unsigned n, const double *x,
double gradient, / NULL if not needed */
void *func_data)

Best,
Richard

[dibyendumajumdar]

Hi Richard,

Thank you for the test case.

Every time the objective function is called, Java arrays are allocated by the calling JNI wrapper. According to the JNI specs these should be released when the JNI function finishes - and therefore GC'd. I will have to find out how this actually behaves - it may be that the arrays are not being collected.

Could you try calling the System.gc() inside the objective function occasionally to see it if makes a difference?

I will look into this issue over the next few days.

Regards
Dibyendu

[ourcookiemonster]

Thanks! Indeed the issue appears to have been around garbage collection of those local JNI java arrays. Unless specifically deleted, the java arrays cannot be GC'd until the native method has finished.

http://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/design.html#global_and_local_references

So despite the "callback" C method completing the "base" JNI method (optimize) was still running ... thereby preventing all of those local JNI java arrays from being GC'd which for a large long-runnuing optimization resulted in OOM.

I added the following 2 lines in nlopt_minfunc to dereference those 2 arrays and recompiled nlopt4j.dll:

jni->DeleteLocalRef(xarray);
jni->DeleteLocalRef(gradientarray);
return result;

I enabled the G1 garbage collector and everything looks good now, memory use completely stable!

-Rich

[dibyendumajumdar]

Fantastic. I will add these calls.

Thank you very much!

[dibyendumajumdar]

Hi Richard

I have committed the fix you suggested. I built the library on Windows 10 using VS2017. But I have not yet been able to test it as the Maven build for the Java components doesn't support this version of Visual Studio. If you are able to test this version then please let me know.

Thanks and Regards
Dibyendu