dose 8 A100 80g enough to finetune 70b llama2 ?

Question

dose 8 A100 80g enough to finetune 70b llama2 ?

james2v opened this issue a year ago · 5 comments

Answer 1 · 2023-08-28T07:42:49.000Z

I think the minimum is 32 * A100 80GB
https://github.com/epfLLM/Megatron-LLM/blob/main/docs/guide/faq.md#what-are-the-basic-hardware-requirements

Answer 2 · 2023-08-28T07:44:30.000Z

I think the minimum is 32 * A100 80GB https://github.com/epfLLM/Megatron-LLM/blob/main/docs/guide/faq.md#what-are-the-basic-hardware-requirements

THANK YOU！ I might tune 70b llama2 with lora then.

Answer 3 · 2023-08-29T01:22:36.000Z

Correct, 32x 80GB is the minimum requirement we have been able to achieve when using sequence length of 4k

Answer 4 · 2023-08-30T08:35:24.000Z

I think the minimum is 32 * A100 80GB https://github.com/epfLLM/Megatron-LLM/blob/main/docs/guide/faq.md#what-are-the-basic-hardware-requirements

THANK YOU！ I might tune 70b llama2 with lora then.

can you run llama2 70b with lora.

Answer 5 · 2023-08-31T04:07:35.000Z

I think the minimum is 32 * A100 80GB https://github.com/epfLLM/Megatron-LLM/blob/main/docs/guide/faq.md#what-are-the-basic-hardware-requirements

THANK YOU！ I might tune 70b llama2 with lora then.

can you run llama2 70b with lora.

i load it in 8 bit to train it. but i use other hud to run it