Solver silently fails when the number of parameters is large, but it's not due to OOM
dmitrijsk opened this issue · 1 comments
Hi, I saw issue #75 about the out-of-memory error issue when the number of parameters is large. In my case, it's not an OOM issue, but ECOS, SCS, OSQP, and SCIPY solvers (I have not tried other solvers) silently* fail and exit. The memory usage is below 50MB, according to the memory profiler. I have 32GB RAM. Here is my minimal example:
import cvxpy as cp
import numpy as np
n_rows = 7200
n_cols = 21
mask_1 = np.zeros(n_rows, dtype=bool)
mask_1[0:3000] = True
mask_2 = np.invert(mask_1)
A = cp.Parameter((n_rows, n_cols), nonneg=True)
d = cp.Variable(n_rows)
t = cp.Variable(n_cols)
constraints = [d == A @ t, t >= 0]
obj_term_1 = cp.pos(1 - d[mask_1])
obj_term_2 = cp.pos(d[mask_2] - 1)
obj_term_3 = cp.sum(t)
objective = cp.Minimize(cp.sum(obj_term_1) + cp.sum(obj_term_2) + obj_term_3)
problem = cp.Problem(objective, constraints)
A.value = np.abs(np.random.randn(n_rows, n_cols))
problem.solve(solver="ECOS", verbose=True)
n_rows = 7200
(the number of rows in the A
parameter), as above, produces the following print-out:
>python script.py
===============================================================================
CVXPY
v1.1.18
===============================================================================
(CVXPY) Apr 23 11:38:23 PM: Your problem has 7221 variables, 2 constraints, and 151200 parameters.
(CVXPY) Apr 23 11:38:23 PM: It is compliant with the following grammars: DCP, DQCP
(CVXPY) Apr 23 11:38:23 PM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.
-------------------------------------------------------------------------------
Compilation
-------------------------------------------------------------------------------
(CVXPY) Apr 23 11:38:23 PM: Compiling problem (target solver=ECOS).
(CVXPY) Apr 23 11:38:23 PM: Reduction chain: Dcp2Cone -> CvxAttr2Constr -> ConeMatrixStuffing -> ECOS
(CVXPY) Apr 23 11:38:23 PM: Applying reduction Dcp2Cone
(CVXPY) Apr 23 11:38:23 PM: Applying reduction CvxAttr2Constr
(CVXPY) Apr 23 11:38:23 PM: Applying reduction ConeMatrixStuffing
And then it silently* exits. If n_rows = 6000
then it solves the problem:
>python script.py
===============================================================================
CVXPY
v1.1.18
===============================================================================
(CVXPY) Apr 23 11:40:57 PM: Your problem has 6021 variables, 2 constraints, and 126000 parameters.
(CVXPY) Apr 23 11:40:57 PM: It is compliant with the following grammars: DCP, DQCP
(CVXPY) Apr 23 11:40:57 PM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.
-------------------------------------------------------------------------------
Compilation
-------------------------------------------------------------------------------
(CVXPY) Apr 23 11:40:57 PM: Compiling problem (target solver=ECOS).
(CVXPY) Apr 23 11:40:57 PM: Reduction chain: Dcp2Cone -> CvxAttr2Constr -> ConeMatrixStuffing -> ECOS
(CVXPY) Apr 23 11:40:57 PM: Applying reduction Dcp2Cone
(CVXPY) Apr 23 11:40:57 PM: Applying reduction CvxAttr2Constr
(CVXPY) Apr 23 11:40:57 PM: Applying reduction ConeMatrixStuffing
(CVXPY) Apr 23 11:41:24 PM: Applying reduction ECOS
(CVXPY) Apr 23 11:42:00 PM: Finished problem compilation (took 6.335e+01 seconds).
(CVXPY) Apr 23 11:42:00 PM: (Subsequent compilations of this problem, using the same arguments, should take less time.)
-------------------------------------------------------------------------------
Numerical solver
-------------------------------------------------------------------------------
(CVXPY) Apr 23 11:42:00 PM: Invoking solver ECOS to obtain a solution.
ECOS 2.0.10 - (C) embotech GmbH, Zurich Switzerland, 2012-15. Web: www.embotech.com/ECOS
It pcost dcost gap pres dres k/t mu step sigma IR | BT
0 -1.319e+00 -1.319e+00 +3e+04 5e-01 6e-01 1e+00 2e+00 --- --- 1 2 - | - -
1 +1.643e+02 +1.663e+02 +1e+04 1e-01 1e-01 2e+00 1e+00 0.7726 3e-01 2 1 2 | 0 0
2 +2.820e+02 +2.825e+02 +2e+03 1e-02 1e-02 5e-01 1e-01 0.8819 3e-02 2 1 2 | 0 0
3 +3.607e+02 +3.608e+02 +3e+02 2e-03 2e-03 8e-02 3e-02 0.8481 3e-02 2 2 2 | 0 0
4 +3.753e+02 +3.753e+02 +6e+01 4e-04 3e-04 1e-02 5e-03 0.8265 5e-02 2 2 2 | 0 0
5 +3.775e+02 +3.775e+02 +3e+01 2e-04 1e-04 5e-03 2e-03 0.6739 1e-01 3 1 2 | 0 0
6 +3.781e+02 +3.781e+02 +2e+01 1e-04 9e-05 2e-03 1e-03 0.4906 3e-01 2 1 1 | 0 0
7 +3.785e+02 +3.785e+02 +9e+00 6e-05 5e-05 1e-03 8e-04 0.5121 1e-01 2 1 1 | 0 0
8 +3.789e+02 +3.789e+02 +3e+00 2e-05 2e-05 2e-04 3e-04 0.8084 2e-01 2 1 1 | 0 0
9 +3.790e+02 +3.790e+02 +8e-01 5e-06 4e-06 5e-05 6e-05 0.7772 2e-02 3 1 1 | 0 0
10 +3.790e+02 +3.790e+02 +2e-01 2e-06 1e-06 2e-05 2e-05 0.7246 7e-02 3 1 1 | 0 0
11 +3.790e+02 +3.790e+02 +8e-02 5e-07 4e-07 4e-06 7e-06 0.7358 7e-02 3 1 1 | 0 0
12 +3.790e+02 +3.790e+02 +8e-03 5e-08 4e-08 1e-07 7e-07 0.9868 9e-02 3 1 1 | 0 0
13 +3.790e+02 +3.790e+02 +1e-04 6e-10 5e-10 2e-09 8e-09 0.9882 1e-04 3 1 1 | 0 0
14 +3.790e+02 +3.790e+02 +1e-06 7e-12 5e-12 2e-11 9e-11 0.9890 1e-04 2 1 1 | 0 0
OPTIMAL (within feastol=6.6e-12, reltol=2.8e-09, abstol=1.1e-06).
Runtime: 0.326742 seconds.
-------------------------------------------------------------------------------
Summary
-------------------------------------------------------------------------------
(CVXPY) Apr 23 11:42:01 PM: Problem status: optimal
(CVXPY) Apr 23 11:42:01 PM: Optimal value: 3.790e+02
(CVXPY) Apr 23 11:42:01 PM: Compilation took 6.335e+01 seconds
(CVXPY) Apr 23 11:42:01 PM: Solver (including time spent in interface) took 3.292e-01 seconds
* About silently. Interestingly, in some cases, when a solver fails, it also crashes PyCharm IDE (even though I run scripts from the Windows CMD) and Google Chrome browser. Once also the screen went black for 1 minute and then restored.
Do you have any suggestion on how to avoid solver crashes? Many thanks in advance!
I noticed that PyCharm reported "insufficient memory for the Java Runtime Environment", so I profiled the memory again. It turns out that profiling a function with memory profiler as python -m memory_profiler script.py
returns incorrect values. When profiled as a script with mprof run script.py
with n_rows = 6000
, it reports 17.5GB RAM used. So it looks like this is a known issue reported in #75.