Megatron-LM’s communication

Question

Closed this issue 6 months ago · 2 comments

Isn't it four allgather and reducescatter？

Answer 1 · 2024-03-18T19:32:37.000Z

There is an extra all-gather here (*2, two block per layer).

Answer 2 · 2024-03-19T11:30:37.000Z

There is an extra all-gather here (*2, two block per layer).

Thank you for your answer. I think I understand it