tlverse/hal9001

centering of sparse design matrix

nhejazi opened this issue · 2 comments

Scaling of the design matrix (of basis functions) provided considerable speedups in our custom implementation of the LASSO -- a ~2x speedup, in fact.

Based on the above observation, it appears that centering of the design matrix might provide further (hopefully considerable) speedups / efficiency gains. That said, it is not immediately clear how to go about efficiently centering sparse matrices. This will require some investigation.

Initial testing shows that this will not decrease lasso computation time. For posterity, the coordinate descent update looks like:

new_beta = resid_sum - sum(resids)*xcenter_j) / n / xscale_j + beta_j;

and the residual update looks like:

int n = X.rows();
    for(int i=0; i<n; ++i){
      resids[i]-=(X.coeff(i,j) - xcenter_j)*beta_diff;
    }

It's possible there might be faster ways to implement these updates, but at least in this form the additional computation required relative to not centering (i.e. the sum in the CD update and the iteration over the zero terms in the residual update) is not worth the decrease in iterations it yields.

Implemented a version that seems to improve fit quality without substantially affecting speed in #23