L1 penalty set small weights to 0
Closed this issue · 9 comments
Dear team,
I have noticed that L1 penalty does not strictly force weights to 0. I wonder what is your view on below tentative approach that forces small weights to strictly 0?
class SmallWeightsToZero(Constraint):
def call(self, w):
mask = tf.abs(w) < tf.keras.backend.epsilon()
w = tf.where(mask, tf.zeros_like(w), w)
return w
model.add(layers.Dense(units = 1,
kernel_regularizer = regularizers.L1(l1=l1),
kernel_constraint = SmallWeightsToZero()))
@coco90417,
Unfortunately with the code provided above, we are unable to debug the issue. Could you please provide the complete standalone code and the tensorflow version you are using. Thank you!
Thank you for looking into this.
Below is what you could reproduce the issue.
import tensorflow as tf
import numpy as np
from tensorflow import keras
1000 observations; 20 predictors, 1 of predictors is redundant of all 0s
so you would be expecting weight for the first predictor to be 0 with L1 penalty
x_training = tf.concat(
[
tf.zeros([1000, 1]),
tf.random.normal([1000, 19], seed = 1),
], axis = 1)
y_training = tf.random.normal([1000], seed = 1)
from keras import models, layers, regularizers, optimizers
lasso regression with tensorflow
model = models.Sequential()
model.add(layers.InputLayer(input_shape = (20, )))
model.add(layers.Dense(units = 1,
kernel_regularizer = regularizers.L1(l1=1000)))
model.compile(optimizer = optimizers.Adam(),
loss = "mse")
fit = model.fit(x_training, y_training, epochs = 1000)
model.get_weights()
[array([[-1.15685923e-06],
[ 5.27249262e-05],
[-1.37503495e-04],
[-1.28477444e-04],
[-2.45119820e-06],
[ 1.45760217e-04],
[ 1.54898667e-04],
[ 2.10382195e-05],
[ 1.31973664e-06],
[-1.33556408e-04],
[ 4.12663186e-06],
[ 1.44984601e-05],
[-1.64269970e-04],
[-1.78692422e-04],
[-1.08664912e-04],
[ 2.58929811e-04],
[-5.15903737e-06],
[ 2.46357895e-05],
[ 7.39442373e-06],
[-1.24024616e-04]]),
array([0.00742478])]
The weight is not even small for predictor 1. The weight constraint was what I thought might address it, but would to hear about your recommended approach.
@coco90417, Unfortunately with the code provided above, we are unable to debug the issue. Could you please provide the complete standalone code and the tensorflow version you are using. Thank you!
Please see my response above. Much appreciated.
@tilakrayal Let me know how may I help on this issue.
Below is output from standard linear regression with L1 penalty, where we can observe shrinking of coefficient to strictly 0 in comparison.
linear regression
from sklearn import linear_model
lr = linear_model.Lasso(alpha=0)
lr.fit(x_training, y_training)
lr.coef_
Out[19]: array([ 0. , -0.05187305, -0.03030285, 0.01014469, 0.05644973,
-0.00163288, -0.01867641, -0.07605389, -0.03441314, 0.03904819,
-0.04583849, -0.02413755, 0.01884066, -0.01466035, 0.02474141,
0.02456712, 0.01741131, 0.0345059 , 0.01402795, 0.02839415])
lasso regression
lr_lasso = linear_model.Lasso(alpha=0.1)
lr_lasso.fit(x_training, y_training)
lr_lasso.coef_
Out[22]: array([ 0., -0., -0., 0., 0., -0., -0., -0., -0., 0., -0., -0., 0.,
-0., 0., 0., 0., 0., 0., 0.])
@sachinprasadhs, kindly follow up on this thread. Please let us know if there is anything we can do to help.
@coco90417,
The L1 penalty in vanilla form doesn't guarantee weights to be exactly zero. While it encourages sparsity by shrinking coefficients towards zero, some small, non-zero values can persist.
This behavior is a key characteristic of L1 regularization and has both advantages and disadvantages depending on the context. The weights not being exactly 0 is an optimisation artifact. If you do linear least squares with L1 regularisation and optimise it with gradient descent you will never reach exactly 0.
Also notice that for L1, the gradient (of the L1 term) is either 1 or -1, except for when w=0. That means that L1-regularization will move any weight towards 0 with the same step size, regardless of the weight’s value. In contrast, you can see that the L2 gradient is linearly decreasing towards 0 as the weight goes towards 0. Therefore, L2-regularization will also move any weight towards 0, but it will take smaller and smaller steps as a weight approaches 0. Thank you!
This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.
This issue was closed because it has been inactive for 28 days. Please reopen if you'd like to work on this further.