QLoRA Finetune Example

Question

QLoRA Finetune Example

sr5434 opened this issue 9 months ago · 11 comments

sr5434 commented 9 months ago

Can you post an example of how to fine-tune a llm with 4-bit QLoRA on a TPU? Thanks for any help you can provide.

sr5434 commented 9 months ago

Nvm

Answer 1 · 2024-02-15T18:53:15.000Z

Yes soon ill add that but what kinda device you are using? Tpu/gpu/xpu?

Answer 2 · 2024-02-15T20:23:18.000Z

I’m using TPU v3

Answer 3 · 2024-02-15T20:23:56.000Z

I just change the Params dtype right?

Answer 4 · 2024-02-15T21:23:12.000Z

you have to use bits=4 or 8 and pass lora config to Xrapture Module

Answer 5 · 2024-02-15T21:30:39.000Z

Ok, thanks

Answer 6 · 2024-02-15T21:35:24.000Z

There doesn’t appear to be an argument for bits

Answer 7 · 2024-02-17T14:59:56.000Z

Nvm

You have to change them inside model config

You can read docs for that or just see examples

Answer 8 · 2024-02-17T15:01:31.000Z

Ok, thanks. Also, is there a way to save just the LoRA adapter in the HuggingFace format after training?

Answer 9 · 2024-02-19T17:38:47.000Z

Im working on that and im gona build that as soon as possible but i have to finish SFTTrainer and make sure that DPOTrainer doesn't have any more bugs.

Answer 10 · 2024-02-19T21:27:36.000Z

Ok, thanks.