[QUESTION]What is the reason why bfloat16 requires gradient accumulation and allreductions need to be completed in fp32

Question

[QUESTION]What is the reason why bfloat16 requires gradient accumulation and allreductions need to be completed in fp32

huhu0823 opened this issue a year ago · 3 comments

Hi,I am currently studying the Megatron framework. I noticed that bfloat16 in megatron requires gradient accumulation, and allreductions need to be completed in fp32. Then the gradient communicates in fp32 format.
However, fp16 requires gradient accumulation, and allreductions can be completed in fp16.Then the gradient communicates in fp16 format.
I want to know what are the special reasons for these two different calculation methods

Line 159-160 of the megatron/arguments. py file

Answer 1 · 2023-11-14T18:19:50.000Z

Marking as stale. No activity in 60 days.

Answer 2 · 2024-02-22T14:26:25.000Z

Same question, anyone knows the reason?

Answer 3 · 2024-04-22T18:20:50.000Z

Marking as stale. No activity in 60 days.