sail-sg/zero-bubble-pipeline-parallelism

Issues

• [QUESTION] IS zero bubble pp support flash-atten?

  #48 opened 4 months ago by qq1243196045
  1

• [QUESTION] fp32_size assert

  #49 opened 4 months ago by liqing9399
  0

• [QUESTION] May I ask what tool was used to plot Figure 6 in paper.How can I profile bubble time in pipeline parallelism?

  #18 opened 9 months ago by starstream
  3

• [ENHANCEMENT] Support selective B-W Split for all schedules

  #32 opened 6 months ago by ufotalent
  1

• [ENHANCEMENT] Opensourcing of 1F1B-V and the variant of interleaved 1F1B and benchmark

  #31 opened 6 months ago by ufotalent
  1

• [ENHANCEMENT] Runtime and scheduler refactor so that we support uniform scheduler API

  #30 opened 6 months ago by ufotalent
  1

• [QUESTION] I used the zero-bubble commit 7ad9c81d for my experiments, and found that the memory usage of this zb-v model exceeds that of the previous zb1 model. What could be the issue? The specific configuration and results are shown in the image.

  #36 opened 6 months ago by lbk-sys
  4

• [QUESTION] Question about ZeroBubbleVPipeScheduler schedule W

  #38 opened 5 months ago by duanjunwen
  1

• [QUESTION] I know that zero-bubble is developed based on Megatron-LM. Could you please let me know which specific commit of Megatron-LM zero-bubble is based on?

  #35 opened 6 months ago by lbk-sys
  1

• [ENHANCEMENT] Sync code base to newest megatron

  #29 opened 6 months ago by ufotalent
  0

• [ENHANCEMENT] Adaptive V scheduler

  #28 opened 6 months ago by ufotalent
  0

• [ENHANCEMENT] Support V-Min and V-Half schedules

  #27 opened 6 months ago by ufotalent
  0

• Per stage memory control

  #8 opened 6 months ago by mavenlin
  1

• [BUG] Doesn't work on older pytorch versions, e.g. 1.14

  #4 opened 6 months ago by ufotalent
  0

• [QUESTION] can ZB work with DeepSpeed (or is there a plan?)

  #23 opened 6 months ago by guoyejun
  5

• [QUESTION] How to avoid synchronization when using sharding

  #19 opened 7 months ago by AndSonder
  3

• [QUESTION] Measuring Pipeline Bubble Time During Megatron-LM Training

  #25 opened 7 months ago by HodBadichi
  1

• [QUESTION]1f1b is fast then zero-v

  #24 opened 7 months ago by kuangdao
  0

• interleaved 1F1B seems to work better

  #21 opened 8 months ago by zhj96
  3

• [QUESTION] The timing for B and W appears to be incorrect

  #22 opened 8 months ago by RookieHong
  2

• More general ZBV scheduling

  #9 opened a year ago by mavenlin
  1

• [QUESTION] Whether to split bw when send_backward_recv_forward is not enabled

  #17 opened 9 months ago by AndSonder
  4

• Support sequence parallel on main branch

  #13 opened a year ago by ufotalent
  1

• Create a miniversion containing only ZB-H1 and essential changes so other megatron forks can easily integrate

  #10 opened a year ago by ufotalent
  5

• [QUESTION] Post-validation Optimizer

  #16 opened a year ago by Parallel-Hao
  1

• Survey on implementing simple version in other codebases besides megatron-lm

  #12 opened a year ago by ufotalent
  0