Possible memory leak in 1.7.0 (nuget, windows)

Question

Possible memory leak in 1.7.0 (nuget, windows)

Closed this issue 4 days ago · 12 comments

We notice a memory leak, using the MIP solver in some .net applications (tested on .net7 and .net8).

A simple test program would be to run multiple MIPs with integer variables (e.g. knapsack) one after another. (see the attached cs file HighsTester_LessDependencies.cs.txt).
Frankly, the leak only appears when using integer constraints on the variables.

The unmanaged memory keeps rising and analysing it shows some log strings from the MIP solver.
We can see that the remaining unmanaged memory contains strings which look like logs of the MIP.

We use the following when working with the solver:

HighsLpSolver solver = new HighsLpSolver();
try
{
	var status = solver.passMip(model);
	status = solver.run();
	if (status == HighsStatus.kError)
	{
		return (null, 0, HighsModelStatus.kUnknown);
	}
	HighsSolution sol = solver.getSolution();
	var info = solver.getInfo();
	HighsModelStatus modelStatus = solver.GetModelStatus();
	return (sol.colvalue, info.ObjectiveValue, modelStatus);
}
finally
{
	solver.clearSolver();
	solver.clearModel();
	solver.Dispose();
}

We haven't been able to test the same in pure C++.

Answer 1 · 2024-05-23T07:45:28.000Z

This is odd. Perhaps you've seen other strings that correspond to MIP output, but they aren't listed here. The strings listed are values assigned as part of the records that are created when the run-time options are set up. It would appear that these records are not cleared when a Highs instance is destroyed

Answer 2 · 2024-05-23T09:35:16.000Z

Here are some further screenshots for strings in unmanaged memory blocks after all solvers have finished.
The program itself requires about 20MB RAM. The test program iterates over 50.000 small knapsack problems (100 variables) and the unmanaged heap blocks 2GB RAM. This test takes just a few seconds. Going up to 1000 variables takes about 5 minutes and increases the leaked memory to 8GB of RAM.

Maybe we are missing some calls to clean up the memory?

Here are the string lists from VMMap of the first three unmanaged memory block of the screenshot above:

Answer 3 · 2024-05-23T09:47:49.000Z

Yes, these are all name and description strings in runtime option records. These are created when the HighsOptions class is instantiated. The destructor for the HighsOptions class calls deleteRecords();,

void deleteRecords() {
for (size_t i = 0; i < records.size(); i++) delete records[i];
}

which loops through all the records. Putting a print statement after delete records[i]; sometimes gives the values for the strings that were defined. I don't know how delete acts - I didn't write deleteRecords();, - and it's beyond my C++ knowledge.

Naively, given the declaration std::vector<OptionRecord*> records;, I'm surprised that there's no records.clear(). I don't see what harm it could do, so I can add it, but it would be good if someone who understands C++ better than me were to give an explanation and fix for what's going on.

Answer 4 · 2024-05-23T10:01:43.000Z

Running valgrind (without records.clear();)gives

==35818==
==35818== HEAP SUMMARY:
==35818== in use at exit: 5,252,312 bytes in 52 blocks
==35818== total heap usage: 6,419 allocs, 6,367 frees, 10,063,469 bytes allocated
==35818==
==35818== LEAK SUMMARY:
==35818== definitely lost: 0 bytes in 0 blocks
==35818== indirectly lost: 0 bytes in 0 blocks
==35818== possibly lost: 5,251,784 bytes in 33 blocks
==35818== still reachable: 528 bytes in 19 blocks
==35818== suppressed: 0 bytes in 0 blocks
==35818== Rerun with --leak-check=full to see details of leaked memory
==35818==
==35818== For lists of detected and suppressed errors, rerun with: -s
==35818== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

When running

valgrind --leak-check=full bin/highs

the leaks correspond to the parallel event scheduler, and I remember a discussion with @gottwald to the effect that they may not be real memory leaks and "don't matter"

If I add the call records.clear();. then valgrind still gives

==35818== still reachable: 528 bytes in 19 blocks

which makes sense if the "memory leaks" on Linux are due to the parallel event scheduler. So, could the issue only relate to Windows?

Answer 5 · 2024-05-24T15:45:43.000Z

We tested our knapsack test application (.Net8) on Ubuntu in wsl.

Running the test application with 500 knapsack problems with 10_000 variables each, the application (console) keeps about 3 GB RAM after all solvers are done, cleared and disposed.

Tracking with valgrind takes ages, so I used a smaller data set with 100 problems and 1000 variables each.

==67673== LEAK SUMMARY:
==67673== definitely lost: 66,472 bytes in 180 blocks
==67673== indirectly lost: 0 bytes in 0 blocks
==67673== possibly lost: 19,682 bytes in 58 blocks
==67673== still reachable: 10,602,507 bytes in 5,864 blocks
==67673== suppressed: 0 bytes in 0 blocks
==67673== Reachable blocks (those to which a pointer was found) are not shown.
==67673== ERROR SUMMARY: 3398 errors from 31 contexts (suppressed: 0 from 0)

The same with only one problem to solve with just one variable gives the following

==70616== HEAP SUMMARY:
==70616== in use at exit: 15,443,039 bytes in 5,349 blocks
==70616== total heap usage: 32,346 allocs, 26,997 frees, 26,152,403 bytes allocated
==70616==
==70616== LEAK SUMMARY:
==70616== definitely lost: 37,276 bytes in 90 blocks
==70616== indirectly lost: 0 bytes in 0 blocks
==70616== possibly lost: 5,275,316 bytes in 105 blocks
==70616== still reachable: 10,130,447 bytes in 5,154 blocks
==70616== suppressed: 0 bytes in 0 blocks

I looks like .net8 and valgrind might not work together that well, at least running in wsl.

@galabovaa Can we provide additional information that might help?

Answer 6 · 2024-05-24T16:24:06.000Z

Are these results with https://github.com/ERGO-Code/HiGHS/tree/latest ?

Answer 7 · 2024-05-24T16:31:08.000Z

The tests use the official nuget package version 1.7.0.

Answer 8 · 2024-05-24T16:36:12.000Z

Sorry, yes, of course.

Since I've found an anomaly relating to the records for run-time options, it's worth seeing whether my fix - that @galabovaa (or someone with better C++ skills than me) will need a view on - clears the issue. However, this will require a new nuget package to be created. This is something that only @galabovaa can do, and she's on holiday until the end of next week.

Answer 9 · 2024-05-25T13:12:53.000Z

I checked our test application on a vm with ubuntu (64 bit), to see if it is a windows related issue.

Running 1000 problems with 1000 variables (weight 0.4, see our example file) starts with 70MB before the first solver is initialized and it ends with 677 MB RAM remaining after all solvers have been disposed.
So, the issue seems to exist on Linux systems as well.
It looks like increasing the knapsack complexity also increases the amount of leaking memory.

It still might be an issue related to the mix of .net and the native highs library.

I took a memory dump and checked for remaining strings after the application disposed all solvers and ran a final aggressive garbage collection (see below).

gcore -o /tmp/core_dump PID

Here are some lines from the command

strings /tmp/core_dump.PID | grep "MIP"

the MIP
pent for MIP heuristics
simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored
the MIP
g MIP solutions should be reported in sparse format
ows in the MIP solver cutpool before they are deleted
f observations before MIP solver pseudo costs are considered reliable
ber of improving solutions found to stop the MIP solver prematurely
it on the number of rows in the MIP solver cutpool for dynamic age adjustment
f observations before MIP solver pseudo costs are considered rel
ng improving MIP solutions: not reported for an empty string ""
f entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi
tion of the MIP solv
MIP
MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi
er when solving LPs, but not subproblems in the MIP solver
ative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance
simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored
e number of rows in the MIP solver cutpool for dynamic age adjustment
the MIP
for termination of the MIP solv
the MIP
simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored
ynamic LP rows before they are removed from the LP relaxation in the MIP solver
the MIP
MIP solutions should b!
MIP in
olute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance
the MIP
the MIP
f entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi
the MIP
the MIP
b|/|ub|, to determine whether optimality has been reached for a MIP instance
lve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive
t in MIP
ber of improving solutions found to stop the MIP solver prematurely
g MIP solutions should be saved
the number of improving solutions found to stop the MIP solver prematur
Minimal number of entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi
b|/|ub|, to determine whether optimality has been reached for a MIP instance
e on absolute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance
number of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive
simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored
e target for termination of the MIP solver
improving MIP solutions should be saved
improving MIP solutions should b
d for a MIP instQ
een reached for a MIP in*
the MIP
f observations before MIP solver pseudo costs are considered reliable
for simplex solver when solving LPs, but not subproblems in the MIP solver
ng MIP
MIP symmetry should be d MIP restart is permitted
ng improving MIP solutions: not reported for an empty string ""
reporting improving MIP solutions: not reported for an empty string ""
e on relative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance
ng improving MIP solutions: not reported for an empty string ""
ber of improving solutions found to stop the MIP solver prematur the MIP( the MIP e number of rows in the MIP solver cutpool for dynamic age adjusP b|/|ub|, to determine whether optimality has been reached for a MIP instance e on absolute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance the number of improving solutions found to stop the MIP solver prematur@ simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored improving MIP solutions should be saved improving MIP solutions should be reported in sparse format the MIP e on relative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored lve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive for simplex solver when solving LPs, but not subproblems in the MIP solver the MIP n in the MIP1 improving MIP solutions should be saved improving MIP solutions should be reported in sparse for
of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive
MIP so!
pent for MIP heuristics
e on absolute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance
f entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi
simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored
e", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored
pent for MIP heuristics
ynamic LP rows before they are removed from the LP relaxation in the MIP solver
age of rows in the MIP solver cutpool before they are de
number of observations before MIP solver pseudo costs are considered reliable
MIP restart is permitted
improving MIP solutions should be reported in sparse format
MIP symmetry should be detected
MIP restart is permitted
for simplex solver when solving LPs, but not subproblems in the MIP solver
ving MIP
MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi
simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored
for termination of the MIP solv@
the MIP
improving MIP solutions should be saved
pent for MIP heuristics
number of observations before MIP solver pseudo costs are considered rel6 age of rows in the MIP solver cutpool before they are de W g MIP solutions should be reported in sparse forP not subproblems in the MIP solv for termination of the MIP solv e on relative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance e on absolute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance the MIP s chosen then, for a MIP (QP) the integr age of rows in the MIP solver cutpool before they are de simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored it on the number of rows in the MIP solver cutpool for dynamic age adjustment ber of improving solutions found to stop the MIP solver prematurely age of rows in the MIP solver cutpool before they are depE e number of rows in the MIP solver cutpool for dynamic age adjus MIP solver cliquetable before neighbour f entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi rom the LP relaxation in the MIP! the MIP e on absolute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance g MIP solutions should be reported in sparse format the MIP age of dynamic LP rows before they are removed from the LP relaxation in the MIP solver e number of rows in the MIP solver cutpool for dynamic age adjus the MIP ng improving MIP solutions: not reported for an empty string "" the MIP the MIP e number of rows in the MIP solver cutpool for dynamic age adjustment ows in the MIP solver cutpool before they are de0 reporting improving MIP solutions: not reported for an empty string "" it on the number of rows in the MIP solver cutpool for dynamic age adjus the MIP the MIP ynamic LP rows before they are removed from the LP relaxation in the MIP solver the MIPC improving MIP solutions should be saved ber of improving solutions found to stop the MIP solver prematurZ
g MIP solutions should be reported in sparse forP
MIP
MIP symmetry should be detected
the MIP
ative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance
olute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance
the MIP
the MIP
age of rows in the MIP solver cutpool before they are deleted
simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored
simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored
for termination of the MIP solver
f entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi
for simplex solver when solving LPs, but not subproblems in the MIP solver
the MIP
MIP heuristics
pent for MIP heuristics
the MIP
the MIP
of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive
number of observations before MIP solver pseudo costs are considered rel MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi the MIP lve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive the MIP the MIPT the MIPU for simplex solver when solving LPs, but not subproblems in the MIP solver er when solving LPs, but not subproblems in the MIP solver Minimal number of entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi age of rows in the MIP solver cutpool before they are dep ative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance olute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance MIP soi improving MIP solutions should be reported in sparse format for termination of the MIP solver the number of improving solutions found to stop the MIP solver prematurely age of rows in the MIP solver cutpool before they are deleted MIP restart is permitted er when solving LPs, but not subproblems in the MIP solver oving MIP soA the MIP the MIP g MIP solutions should be saved ynamic LP rows before they are removed from the LP relaxation in the MIP solver MIP symmetry should be d
improving MIP solutions should be saved
ng improving MIP solutions: not reported for an empty string ""
number of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive
f entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi pent for MIP heuristics simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored e on relative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance reporting improving MIP solutions: not reported for an empty string "" g MIP solutions should be saved e on relative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance efore they are removed from the LP relaxation in the MIP solver ving MIP0 for simplex solver when solving LPs, but not subproblems in the MIP solver number of observations before MIP solver pseudo costs are considered rel number of observations before MIP solver pseudo costs are considered rel ynamic LP rows before they are removed from the LP relaxation in the MIP solver it on the number of rows in the MIP solver cutpool for dynamic age adjus number of observations before MIP solver pseudo costs are considered rel Minimal number of entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi it on the number of rows in the MIP solver cutpool for dynamic age adjus
MIP symmetry should be d MIP restart is permitted improving MIP solutions should be reported in sparse for MIP restart is permittedp e target for termination of the MIP solv of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive improving MIP solutions should be saved MIP symmetry should be d
the MIPl
the MIPm
age of dynamic LP rows before they are removed from the LP relaxation in the MIP solver
, |ub-lb|, to determine whether optimality has been reached for a MIP instance
e on absolute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance
pent for MIP heuristics
simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored
MIP
e", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored
number of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive
MIP symmetry should be d MIP restart is permitted improving MIP solutions should be saved of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive Solver option: "simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored ber of improving solutions found to stop the MIP solver prematur
the MIP
the MIP
, |ub-lb|, to determine whether optimality has been reached for a MIP instance
simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored
e", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored
MIP symmetry sho
number of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive
...

Answer 10 · 2024-05-25T17:11:29.000Z

Yes, as before, all these are from the records for run-time options, that may now be cleared as a consequence of my fix yesterday, but we can't confirm until a new NuGet package is created. This can't happen until @galabovaa is back

Thanks for your patience

Answer 11 · 2024-06-12T11:50:49.000Z

We have now created a new release on NuGet, containing all of our latest fixes. @Marluwe, would you please give it a go, and let us know if you are still getting memory issues?

Answer 12 · 2024-06-29T12:58:47.000Z

The new version doesn't show any leaks in our tests.
Thank you for the fix!