How to optimize model fine-tuning if it is very slow? (NVIDIA GeForce RTX 4060 Laptop GPU)

Question

How to optimize model fine-tuning if it is very slow? (NVIDIA GeForce RTX 4060 Laptop GPU)

mack007liu opened this issue 9 months ago · 3 comments

mack007liu commented 9 months ago

Answer 1 · 2023-12-20T01:26:37.000Z

mack007liu commented 9 months ago

Answer 2 · 2023-12-20T01:27:02.000Z

mack007liu commented 9 months ago

Answer 3 · 2024-01-03T20:47:00.000Z

Hi @mack007liu thanks for reporting the issue.
The slow training is mostly due to low GPU memory. Mistral-7b + QLora trained on the toy dataset consumes 17GB GPU memory. You may try a smaller model like Phi-1.5 on your laptop.

Below are training time measured a few GPUs (listed in the release notes)