cvxgrp/cvxpylayers

Solver silently fails when the number of parameters is large, but it's not due to OOM

dmitrijsk opened this issue · 1 comments

Hi, I saw issue #75 about the out-of-memory error issue when the number of parameters is large. In my case, it's not an OOM issue, but ECOS, SCS, OSQP, and SCIPY solvers (I have not tried other solvers) silently* fail and exit. The memory usage is below 50MB, according to the memory profiler. I have 32GB RAM. Here is my minimal example:

import cvxpy as cp
import numpy as np


n_rows = 7200
n_cols = 21

mask_1 = np.zeros(n_rows, dtype=bool)
mask_1[0:3000] = True
mask_2 = np.invert(mask_1)

A = cp.Parameter((n_rows, n_cols), nonneg=True)
d = cp.Variable(n_rows)
t = cp.Variable(n_cols)
constraints = [d == A @ t, t >= 0]

obj_term_1 = cp.pos(1 - d[mask_1])
obj_term_2 = cp.pos(d[mask_2] - 1)
obj_term_3 = cp.sum(t)

objective = cp.Minimize(cp.sum(obj_term_1) + cp.sum(obj_term_2) + obj_term_3)
problem = cp.Problem(objective, constraints)
A.value = np.abs(np.random.randn(n_rows, n_cols))
problem.solve(solver="ECOS", verbose=True)

n_rows = 7200 (the number of rows in the A parameter), as above, produces the following print-out:

>python script.py
===============================================================================
                                     CVXPY
                                    v1.1.18
===============================================================================
(CVXPY) Apr 23 11:38:23 PM: Your problem has 7221 variables, 2 constraints, and 151200 parameters.
(CVXPY) Apr 23 11:38:23 PM: It is compliant with the following grammars: DCP, DQCP
(CVXPY) Apr 23 11:38:23 PM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.
-------------------------------------------------------------------------------
                                  Compilation
-------------------------------------------------------------------------------
(CVXPY) Apr 23 11:38:23 PM: Compiling problem (target solver=ECOS).
(CVXPY) Apr 23 11:38:23 PM: Reduction chain: Dcp2Cone -> CvxAttr2Constr -> ConeMatrixStuffing -> ECOS
(CVXPY) Apr 23 11:38:23 PM: Applying reduction Dcp2Cone
(CVXPY) Apr 23 11:38:23 PM: Applying reduction CvxAttr2Constr
(CVXPY) Apr 23 11:38:23 PM: Applying reduction ConeMatrixStuffing

And then it silently* exits. If n_rows = 6000 then it solves the problem:

>python script.py
===============================================================================
                                     CVXPY
                                    v1.1.18
===============================================================================
(CVXPY) Apr 23 11:40:57 PM: Your problem has 6021 variables, 2 constraints, and 126000 parameters.
(CVXPY) Apr 23 11:40:57 PM: It is compliant with the following grammars: DCP, DQCP
(CVXPY) Apr 23 11:40:57 PM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.
-------------------------------------------------------------------------------
                                  Compilation
-------------------------------------------------------------------------------
(CVXPY) Apr 23 11:40:57 PM: Compiling problem (target solver=ECOS).
(CVXPY) Apr 23 11:40:57 PM: Reduction chain: Dcp2Cone -> CvxAttr2Constr -> ConeMatrixStuffing -> ECOS
(CVXPY) Apr 23 11:40:57 PM: Applying reduction Dcp2Cone
(CVXPY) Apr 23 11:40:57 PM: Applying reduction CvxAttr2Constr
(CVXPY) Apr 23 11:40:57 PM: Applying reduction ConeMatrixStuffing
(CVXPY) Apr 23 11:41:24 PM: Applying reduction ECOS
(CVXPY) Apr 23 11:42:00 PM: Finished problem compilation (took 6.335e+01 seconds).
(CVXPY) Apr 23 11:42:00 PM: (Subsequent compilations of this problem, using the same arguments, should take less time.)
-------------------------------------------------------------------------------
                                Numerical solver
-------------------------------------------------------------------------------
(CVXPY) Apr 23 11:42:00 PM: Invoking solver ECOS  to obtain a solution.

ECOS 2.0.10 - (C) embotech GmbH, Zurich Switzerland, 2012-15. Web: www.embotech.com/ECOS

It     pcost       dcost      gap   pres   dres    k/t    mu     step   sigma     IR    |   BT
 0  -1.319e+00  -1.319e+00  +3e+04  5e-01  6e-01  1e+00  2e+00    ---    ---    1  2  - |  -  -
 1  +1.643e+02  +1.663e+02  +1e+04  1e-01  1e-01  2e+00  1e+00  0.7726  3e-01   2  1  2 |  0  0
 2  +2.820e+02  +2.825e+02  +2e+03  1e-02  1e-02  5e-01  1e-01  0.8819  3e-02   2  1  2 |  0  0
 3  +3.607e+02  +3.608e+02  +3e+02  2e-03  2e-03  8e-02  3e-02  0.8481  3e-02   2  2  2 |  0  0
 4  +3.753e+02  +3.753e+02  +6e+01  4e-04  3e-04  1e-02  5e-03  0.8265  5e-02   2  2  2 |  0  0
 5  +3.775e+02  +3.775e+02  +3e+01  2e-04  1e-04  5e-03  2e-03  0.6739  1e-01   3  1  2 |  0  0
 6  +3.781e+02  +3.781e+02  +2e+01  1e-04  9e-05  2e-03  1e-03  0.4906  3e-01   2  1  1 |  0  0
 7  +3.785e+02  +3.785e+02  +9e+00  6e-05  5e-05  1e-03  8e-04  0.5121  1e-01   2  1  1 |  0  0
 8  +3.789e+02  +3.789e+02  +3e+00  2e-05  2e-05  2e-04  3e-04  0.8084  2e-01   2  1  1 |  0  0
 9  +3.790e+02  +3.790e+02  +8e-01  5e-06  4e-06  5e-05  6e-05  0.7772  2e-02   3  1  1 |  0  0
10  +3.790e+02  +3.790e+02  +2e-01  2e-06  1e-06  2e-05  2e-05  0.7246  7e-02   3  1  1 |  0  0
11  +3.790e+02  +3.790e+02  +8e-02  5e-07  4e-07  4e-06  7e-06  0.7358  7e-02   3  1  1 |  0  0
12  +3.790e+02  +3.790e+02  +8e-03  5e-08  4e-08  1e-07  7e-07  0.9868  9e-02   3  1  1 |  0  0
13  +3.790e+02  +3.790e+02  +1e-04  6e-10  5e-10  2e-09  8e-09  0.9882  1e-04   3  1  1 |  0  0
14  +3.790e+02  +3.790e+02  +1e-06  7e-12  5e-12  2e-11  9e-11  0.9890  1e-04   2  1  1 |  0  0

OPTIMAL (within feastol=6.6e-12, reltol=2.8e-09, abstol=1.1e-06).
Runtime: 0.326742 seconds.

-------------------------------------------------------------------------------
                                    Summary
-------------------------------------------------------------------------------
(CVXPY) Apr 23 11:42:01 PM: Problem status: optimal
(CVXPY) Apr 23 11:42:01 PM: Optimal value: 3.790e+02
(CVXPY) Apr 23 11:42:01 PM: Compilation took 6.335e+01 seconds
(CVXPY) Apr 23 11:42:01 PM: Solver (including time spent in interface) took 3.292e-01 seconds

* About silently. Interestingly, in some cases, when a solver fails, it also crashes PyCharm IDE (even though I run scripts from the Windows CMD) and Google Chrome browser. Once also the screen went black for 1 minute and then restored.

Do you have any suggestion on how to avoid solver crashes? Many thanks in advance!

I noticed that PyCharm reported "insufficient memory for the Java Runtime Environment", so I profiled the memory again. It turns out that profiling a function with memory profiler as python -m memory_profiler script.py returns incorrect values. When profiled as a script with mprof run script.py with n_rows = 6000, it reports 17.5GB RAM used. So it looks like this is a known issue reported in #75.