kongds/MoRA

support QMoRA?

Opened this issue · 1 comments

ByungKwanLee commented

is it possible to support QMoRA with huggingface bitsandbytes?

kongds commented

Thanks for your interest in our work.

MoRA has been implemented with Linear4bit in peft-mora is bnb.py, so it can run with a 4-bit model like LoRA. (But I haven't tried this before, so there may be some problems.)