google/paxml

LoRA support in transformers

tanmayshishodia opened this issue · 1 comments

tanmayshishodia commented

Hi I went through the praxis.transformers.StackedTransformer layer and I don't see any support for LoRA.

That said I was wondering if there was a way to add a set of new LoRA weights to an already existing paxConfig model. If you could give any example that would be great. The tutorials don't cover any such use case where we can update the model layers later.

tanmayshishodia commented

#83

Refer above PR if you are looking to implement LoRA in PaxML for now.