model weight without training
BaohaoLiao opened this issue · 4 comments
Is it possible to release different bit model weights without any training? I.e. newly initialized LoRA weights and Q.
Good suggestion! We don't have them now, but it's totally possible to do them on your own --- we recently released the artifacts you will need to do them (with the bits of your choice, though only 64-rank for now). If you bump into any question, feel free to send questions here, and I'm happy to answer!
AFAIK, this should be fairly straightforward + fast. The main caveat is that, for the 70B model case, you will need enough "CPU RAM" (not GPU RAM) to temporarily load the model weights + artifacts --- something like in the order of ~300GB.
May I ask how you get the answer for QLoRA 3-bit? If I remember correctly, QLoRA only supports 4bit and 8bit.
We implement our own version of QLoRA, and hence we can vary the bits.
Closing this assuming the question is addressed. Please feel free to re-open if you have follow-up questions.