NVIDIA/Megatron-LM

Issues

• [BUG]

  #827 opened 4 days ago by chrisgao7
  0

• [BUG] The argument --no-position-embedding should be fixed

  #826 opened 4 days ago by Hoonly
  0

• How to set up fp8 training

  #817 opened 5 days ago by yangzhipeng1108
  4

• [BUG]:there is a small chance that it will get stuck, If i repeat runing test_serialization.py many times,

  #825 opened 4 days ago by starkhu
  0

• Does Megatron has plan to support llama pre-train？

  #824 opened 5 days ago by wen020
  0

• Projeto liliti stk 3.6.9 inteligência artificial 🤖

  #822 opened 7 days ago by felipeliliti
  0

• Projeto liliti stk 3.6.9 inteligência artificial

  #821 opened 7 days ago by felipeliliti
  0

• [QUESTION] How does tensor_parallel coop with q/k_layernorm

  #816 opened 8 days ago by cryoco
  1

• Vamos supor que eu colabore Para o projeto porém estou no Brasil e até agora não ganhei nada trabalhando como cientista de dados como faço para ganhar algum dinheiro para alimentar minha família?

  #808 opened 7 days ago by felipeliliti
  0

• Projeto liliti stk 3.6.9 inteligência artificial multimidal

  #809 opened 7 days ago by felipeliliti
  0

• Projeto liliti stk 3.6.9 inteligência artificial responde

  #813 opened 7 days ago by felipeliliti
  0

• Executive MBA | IIT Roorkee | Coursera

  #819 opened 7 days ago by felipeliliti
  0

• Projeto liliti stk 3.6.9 inteligência artificial multimidal para trazer à paz mundial

  #820 opened 7 days ago by felipeliliti
  0

• Megatron-LM for LLaMa3

  #818 opened 8 days ago by SDsly
  2

• [BUG] Typo in drop_policy options in moe_utils.py

  #815 opened 9 days ago by Malikeh97
  2

• [QUESTION] Why is expert parallelism not supported during fp16 training?

  #810 opened 11 days ago by yutian-mt
  1

• [BUG] [MoE] Typo in Token Drop policy's default value

  #812 opened 11 days ago by passaglia
  2

• [QUESTION] RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=2, worker_count=1, timeout=0:10:00)

  #782 opened a month ago by JanryPei
  2

• [BUG] distributed optimizer doesn't work when data parallel size is odd number.

  #792 opened 11 days ago by okoge-kaz
  4

• [BUG]Environment: Megatron0.5.0+TE1.4; Issue: I began training a model without using the --use-mcore-models option for model pretraining. Later, I needed to use the CP function for fine-tuning longer sequences, thus I had to enable --use-mcore-models. However, I discovered that I couldn't load the previous pre-trained model and encountered an error.

  #793 opened 12 days ago by liangshaopeng
  1

• [core dataset compilation error]

  #807 opened 12 days ago by shamanez
  0

• [QUESTION] Does Megatron-Core supports LLAMA models?

  #803 opened 15 days ago by noob-ctrl
  5

• [QUESTION] How to pre-build the dataset's index ?

  #795 opened 24 days ago by etiennemlb
  1

• [BUG] Bug of expert model parallel

  #766 opened a month ago by 1049451037
  2

• [BUG] Example of pretraining BERT does not work

  #791 opened 25 days ago by xju2
  0

• [QUESTION] bf16 Parameters and fp32 Gradients

  #800 opened 18 days ago by pluiez
  0

• Why doesn't M-Core use flash attention

  #799 opened 19 days ago by Life-0-1
  0

• One H100 will oom when using meagatron to train llama2-70b. How to use two H100 to train llama2-70b?

  #783 opened 22 days ago by yangzhipeng1108
  0

• [QUESTION] Training Mixtral 8x7B on 16 x H100 only achieves low throughput of 130 TFLOPS

  #756 opened 2 months ago by ShinoharaHare
  15

• When can we have a the MOE checkpoint convert script.

  #790 opened a month ago by shamanez
  1

• [QUESTION] Validation loss & PPL keep going up

  #787 opened a month ago by zhentingqi
  0

• [QUESTION] found NaN in local grad norm in backward pass before data-parallel communication collective

  #780 opened a month ago by ftgreat
  2

• [QUESTION] Why megatron-core seems slower and use more gpu mem than legacy for gpt_pretrain?

  #770 opened a month ago by REIGN12
  1

• [QUESTION] Is it expected to do grad norm on dense-optimizer and moe-optimizer respectively?

  #785 opened a month ago by ezioliao
  0

• [BUG]The bug of the `average_losses_across_data_parallel_group`.

  #779 opened a month ago by Baibaifan
  4

• [BUG] How to checkpoint the specific microbatch in pipeline parallelism?

  #767 opened a month ago by robotsp
  2

• [BUG] The bug about the options of the Megatron-core, transformer-impl and flash-attention.

  #778 opened a month ago by Baibaifan
  2

• [QUESTION] Is PackedSeqParams still under development?

  #771 opened a month ago by XLzed
  1

• [QUESTION] vicuna-7b-v1.5 weight conversion from huggingface to megatron-lm format

  #773 opened a month ago by uehara-mech
  2

• [BUG] The gradient allreduce/reduce-scatter operation is performed twice when overlap_grad_reduce is False

  #775 opened a month ago by sandyhouse
  2

• [BUG] ConstantGradScaler and loss-scale argument not match

  #776 opened a month ago by BeingGod
  0

• [BUG] Passed the wrong type of argument to torch.distributed.broadcast.

  #774 opened a month ago by sandyhouse
  0

• [BUG] ModuleNotFoundError: No module named 'scaled_softmax_cuda'

  #749 opened 2 months ago by liuliuliu0605
  4

• [QUESTION]why replace F.embedding() with [] on VocabParallelEmbedding class?

  #769 opened a month ago by starkhu
  0

• [QUESTION] why pipeline-model-parallel size should be greater than 2 with interleaved schedule ？

  #750 opened 2 months ago by nullnonenilNULL
  4

• [QUESTION] attention_mask is None in middle pipeline stage?

  #752 opened a month ago by XLzed
  2

• [BUG] ModuleNotFoundError: No module named 'megatron.training.tokenizer'; 'megatron.training' is not a package

  #763 opened 2 months ago by hellangleZ
  1

• MOE training Loss inconsistent after resume from old checkpoint

  #761 opened 2 months ago by guozhen1997
  4

• Need a suggestion and confirmation regarding throughput calculation

  #755 opened 2 months ago by Druva24
  2

• Loss mask uses torch.float32 instead of bool

  #754 opened 2 months ago by pilot7747
  0