Merge LoRA into 405B

Question

Merge LoRA into 405B

junzhang-zj opened this issue 3 months ago · 4 comments

I am blocked from merging LoRA into LLaMA-3.1-405B with INT8 of BNB, and the specific details are below.
Is there any action that I can try?
huggingface/peft#2065 (comment)

Answer 1 · 2024-09-16T10:59:08.000Z

Error positioning: peft/utils/integrations.py
im, imt, SCim, SCimt, coo_tensorim = bnb.functional.double_quant(im)

Answer 2 · 2024-09-16T14:36:34.000Z

Hi @junzhang-zj Can you let us know what versions of bitsandbytes, transformers, and PEFT that you are using?

Answer 3 · 2024-09-17T02:53:19.000Z

@matthewdouglas I have tried these versions bitsandbytes (0.43.3), transformers (4.44.2/4.43.3), and peft (0.12.0).

Answer 4 · 2024-09-19T01:55:55.000Z

Any updates on this issue?