alibaba/Megatron-LLaMA

Issues

• megatron-lm中的DistributedOptimizer类似deepspeed的zero2吗？梯度切分部分求大佬上源码。

  #71 opened a month ago by zhoushuang66
  0

• 执行hf转megatron格式报错了

  #70 opened 3 months ago by Lilypad97
  0

• TypeError: perform_nocolor_split(): incompatible function arguments.

  #69 opened 5 months ago by vedantgoswami
  0

• When training BERT, ERROR: "AttributeError: 'FullTokenizer' object has no attribute 'save_pretrained' "

  #68 opened 5 months ago by yuzhiguo07
  0

• No module named 'megatron.tokenizer.file_utils'

  #67 opened 5 months ago by yuzhiguo07
  0

• 如何断点续训

  #66 opened 6 months ago by MAxx8371
  0

• No update for a long time

  #65 opened 7 months ago by dong-liuliu
  0

• Has llama2 GQA been supported yet?

  #64 opened 7 months ago by JiwenJ
  0

• 请问有dingding群聊，或者微信群吗？可以沟通的

  #63 opened 8 months ago by felix0080
  0

• Llama 3 Support

  #62 opened 8 months ago by john-theo
  0

• 请问是否支持从0训练一个小规模的LLaMA模型，如：1B

  #59 opened 10 months ago by liubo12
  1

• llama中decoder layer层里面的MLP问题

  #50 opened a year ago by yuanzhoulvpi2017
  4

• Unable to import Megatron

  #51 opened a year ago by fyf2016
  8

• 注意力权重转换问题

  #58 opened a year ago by noob-ctrl
  2

• About batch_size

  #61 opened 10 months ago by tszslovewanpu
  0

• Megatron-LM权重转hf格式

  #52 opened a year ago by Yang-QW
  4

• sh LLaMA2_7B_standalone.sh

  #60 opened 10 months ago by yangzhipeng1108
  0

• 请问目前Megatron-LLaMA支持LLaMA2-70B的训练吗？

  #45 opened a year ago by 13416157913
  1

• 在模型转换权重时遇到了如下问题 Zarr-based strategies will not be registered because of missing packages

  #57 opened a year ago by ZhangEnmao
  0

• llama2-34b shape不匹配

  #21 opened a year ago by cdj0311
  4

• 使用distributed optimzer时grad_norm计算准确度的疑问

  #56 opened a year ago by chivychao
  1

• LLaMAModel._causal_lm_process中的labels和logits对齐方法疑问

  #53 opened a year ago by chivychao
  3

• 出现 forward() missing 1 required positional argument: 'memory_efficient'

  #55 opened a year ago by TongLi3701
  0

• 问下readme中32机的吞吐对应的参数可以提供下吗，目前没有复现出来

  #49 opened a year ago by jianzi123
  5

• 求一份Serving的教程代码

  #48 opened a year ago by xealml
  1

• hf权重转换代码小bug

  #47 opened a year ago by yuanzhoulvpi2017
  0

• 大家好，请教一个关于GLOBAL_BATCH_SIZE值计算的问题，希望大家不吝赐教。

  #35 opened a year ago by 13416157913
  1

• INT4 量化的模型可以被Megatron-LLaMA支持吗？

  #46 opened a year ago by Jeff123z
  1

• 对于不同参数模型，如何通过配置参数信息计算显存占用大小？

  #36 opened a year ago by 13416157913
  1

• 每次GA的backward都需要做通信

  #42 opened a year ago by jingjie01ai
  5

• 是否兼容sequence parallel

  #44 opened a year ago by jingjie01ai
  2

• CUDA_DEVICE_MAX_CONNECTIONS 设置问题

  #43 opened a year ago by Richie-yan
  0

• 请教下为什么使用overlapped_distributed_optimizer后，CUDA_DEVICE_MAX_CONNECTIONS就可以不为1了？

  #26 opened a year ago by yinzhijian
  5

• fp16的支持问题

  #41 opened a year ago by XUWeijiang
  1

• TypeError: OverlappedDistributedOptimizer.gather_parameters() got an unexpected keyword argument 'skip_if_not_stepped'

  #40 opened a year ago by Double-bear
  4

• OverlappedDistributedOptimizer 支持 pipeline parallelism > 1 和 data parallelism > 1 同时使用吗？

  #37 opened a year ago by Baibaifan
  8

• pipeline parallel fwd/bwd里面为什么没有调用optimizer.backward_epilogue()

  #29 opened a year ago by jingjie01ai
  4

• 训练LLaMA2-70B模型，4个节点，A800*80GB 设置8张量并行，1流水线并行，训练报错

  #34 opened a year ago by 13416157913
  0

• Loss对齐

  #31 opened a year ago by wuziyou199217
  3

• 多节点训练时使用nccl后端，在训练完后，保存检查点时报错

  #32 opened a year ago by 13416157913
  1

• 在A800*8卡的机器上，开启 overlapped-distributed-optimizer 的速度比开启 use-distributed-optimizer 的慢约8%

  #22 opened a year ago by tingkuanpei
  6

• 训练llama-30b模型报错是不支持llama-30b模型么？

  #30 opened a year ago by 13416157913
  0

• 运行时无法找到fused_kernels/build/scaled_upper_triang_masked_softmax_cuda.so

  #28 opened a year ago by xikaluo
  5

• 4台A100*8测试，开启 overlapped-distributed-optimizer 的速度比开启 use-distributed-optimizer 慢很多

  #27 opened a year ago by silingtong123
  2

• nccl通信边界问题？

  #17 opened a year ago by Baibaifan
  10

• 请问ParameterSchedule实际上有作用吗？

  #25 opened a year ago by yinzhijian
  1

• 请教一下，怎么感觉LLaMA2-7B模型单机A800*8*80G 用8张卡预训练TP4-PP1-DP2时间和TP1-PP1-DP8时间不合理

  #24 opened a year ago by 13416157913
  1

• NGC22.08 环境报错。

  #23 opened a year ago by EthanChen1234
  2

• 训练完后，将保存的Megatron格式转成HF格式报错

  #20 opened a year ago by 13416157913
  7

• deepspeed+megatron+llama，请问作者有试过吗

  #19 opened a year ago by Chandler-Bing
  1