/megatron-lm-parallel-group-playground

Primary LanguagePython

我也维护了一个cuda以及LLM学习仓库 https://github.com/BBuf/how-to-optim-algorithm-in-cuda , 有需要的小伙伴可以点一点star

Megatron-LM Parallel Group Playground

This is a playground for parallelism group display in Megatron-LM.

1. requirements

  • gradio>=4.16.0

2. deploy

python3 app.py

then open the link created by gradio.

You can also visit huggingface space link .

3. display example

图片

Plan

  • 2D Parallelism Group support (DP+TP, DP+PP)
  • 3D Parallelism Group support (DP+TP+PP)
  • 3D Parallelism Group support (CP+DP, CP+TP, CP+PP, CP+TP+PP)
  • Zero
  • Collective Communication Display

contributions and issues are welcome.

Reference