xrsrke/pipegoose

Readme
Issues
Stargazers
Watchers

Deparallelize pipeline parallelism

xrsrke opened this issue a year ago · 0 comments

xrsrke commented a year ago

Deparallelize pipeline parallelism

Topics

megatron megatron-lm transformers 3d-parallelism data-parallelism pipeline-parallelism tensor-parallelism model-parallelism zero-1 large-scale-language-modeling huggingface-transformers distributed-optimizers mixture-of-experts moe sequence-parallelism

Share to

Contact site admin: Geeks.