L0 regularization in AdapterBias

Question

L0 regularization in AdapterBias

Opened this issue a year ago · 2 comments

Hello, I can't understand your L0 regularization in AdapterBias section after reading the paper, and I can't find the code. Could you please send me the code of L0 regularization in AdapterBias?

Answer 1 · 2023-09-22T03:30:45.000Z

我也想知道

Answer 2 · 2023-09-22T15:11:16.000Z

Hi,
Our implementation of L0 regularization is the same as diff-pruning(https://arxiv.org/abs/2012.07463). The code could be found in here(https://github.com/dguo98/DiffPruning/blob/main/examples/run_glue_diffpruning.py). However, the performance of utilizing L0 regularization on AdapterBias did not work well. We recommend directly fine-tuning on AdapterBias.