When fine-tuning llama-7b, approximately how much GPU memory is required for training?

Question

When fine-tuning llama-7b, approximately how much GPU memory is required for training?

zty07 opened this issue a year ago · 4 comments

When fine-tuning llama, approximately how much GPU memory is required for training?

Answer 1 · 2024-02-13T17:04:36.000Z

Hi @zty07, sorry for the extremely late response. Could you please clarify which experiment you are interested in running? The memory would depend on the task (specifically the sequence length). The quantization code is somewhat broken unfortunately but will be fixed soon which should help with lowering the memory requirements.

Answer 2 · 2024-03-21T03:35:36.000Z

@MJ10 Did the quantization code ever get fixed?

Answer 3 · 2024-04-28T21:04:14.000Z

@MJ10 -- running the next sentence code with 2B/3B parameter size model thrown OOM? any suggestion to resolve?
(PS: I used A100 80GB 8GPUs)

Answer 4 · 2024-05-14T04:56:29.000Z

@abdalgader-a I have managed to get it running on a single A100, but my num_samples is way less than 20