PiSSA Updates Base Model Weights Silently

System Info

Peft v0.13.0
Transformers v4.46.0

Who can help?

Since this relates to an interaction with PEFT maybe @BenjaminBossan @sayakpaul?

Information

The official example scripts
My own modified scripts

Tasks

An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
My own task or dataset (give details below)

Reproduction

Load any model
Load a trained LoRA adapter that was trained without PiSSA
Load a trained LoRA adapter that was trained with PiSSA
Run inference and see that now if you set_adapter to the first adapter, it will output garbage

This is because PiSSA silently updates the weights of the base model and there is nothing checking what else might be impacted by this operation

peft/src/peft/tuners/lora/layer.py

Line 249 in b3176ef

self.get_base_layer().weight.data = weight

model_dir = "/path/to/seq2seq/model"
tokenizer = AutoTokenizer.from_pretrained(model_dir)
model:PeftAdapterMixin = AutoModelForSeq2SeqLM.from_pretrained(model_di).to(device)
profiles = {
    "non-pissa" : "path/to/model/trained/without/pissa",
    "pissa": "path/to/model/trained/with/pissa"
}
for profile in profiles:
    adapter_path = profiles[profile]
    adapter_name = profile
    model.load_adapter(adapter_path, adapter_name)
### Running inference on 'non-pissa' will output garbage because 'pissa' updated the base model weights

Expected behavior

I'm not sure if this ticket belongs here or in the PEFT lib.

I think there are a few options:

a. Throw an error if trying to load a PiSSA adapter when another adapter is loaded, or throw an error if trying to load a non-pissa adapter after a pissa adapter has been loaded
b. Log a warning message that PiSSA updates the base model weights
c. Update PEFT documentation to make it clear that PiSSA updates base model weights and cannot be combined with other adapters. (In this case this ticket probably belongs in the PEFT lib)
d. something else?

Hey, thanks for opening the issue. I "stole" it for PEFT as it's more a PEFT issue :)

The problem you describe is indeed accurate. This is why we added a way to convert a PiSSA adapter to a normal LoRA adapter: https://github.com/fxmeng/peft/tree/main/examples/pissa_finetuning#convert-pissa-to-lora. Could you please check out if this solves the issue for you?

Apart from that, we can still think about giving some type of warning if the situation occurs, but I'm not completely sure how to detect this reliably, I'd have to think about it.

Thanks! I think the main thing is that it would be good to warn a user if an adapter isn't compatible. I had to figure it out via a lot of debugging because there were no warnings, just all the sudden one of my adapters (the non Pissa one) started outputting gibberish and it wasn't clear why. We use adapters under the assumption that they operate independent of each other when you swap between them.