OpenLMLab/LOMO

Issues

• 请问adalomo可以支持用transformer中的trainer训练么？或者未来有可能实现么？

  #73 opened 5 months ago by lyt719
  20

• One confusion about the LOMO paper

  #79 opened 7 months ago by Pairshoe
  1

• wandb permission

  #23 opened a year ago by season1blue
  4

• MOE and Custom Finetuning

  #77 opened 8 months ago by DRXD1000
  0

• installable package

  #74 opened 9 months ago by Borda
  2

• Serious conclusion: LOMO does not significantly reduce GPU memory usage！

  #72 opened a year ago by misonsky
  17

• 为什么LOMO并没有火起来呢？

  #47 opened a year ago by Flywolfs
  5

• eval environment for opencompass

  #65 opened a year ago by KaiLv69
  4

• How to load a 65B model on 24G GPU memory?

  #71 opened a year ago by misonsky
  37

• 实测LOMO++ deepspeed zero2 7b qlora llama 显存占用 感觉比正常的qlora + deepspeed zero2 显存占用大1倍

  #70 opened a year ago by zlh1992
  1

• Can you provide detailed dependency versions?

  #69 opened a year ago by misonsky
  0

• Evaluation of Fine-tuned model with Adalomo

  #62 opened a year ago by sglucas
  1

• adalomo在使用chatglm2模型出现错误

  #68 opened a year ago by JorunoJobana
  2

• adalomo 试图在 llama 2 70b 模型训练时出现 NCCL communicator 这类超时的错误

  #67 opened a year ago by HappyLynn
  5

• Reproduce the results for LOMO

  #66 opened a year ago by shawnricecake
  2

• adalomo optimizer error

  #63 opened a year ago by shawnricecake
  15

• Instructions for evaluation datasets

  #64 opened a year ago by KaiLv69
  3

• model merge error

  #59 opened a year ago by shawnricecake
  18

• Mistral Support

  #58 opened a year ago by freegheist
  1

• type object 'torch._C._distributed_c10d.ReduceOp' has no attribute 'AVG'

  #32 opened a year ago by season1blue
  4

• ModuleNotFoundError: No module named 'rich' after ' python -m pip install rich'

  #24 opened a year ago by SeekPoint
  1

• cannot find adalomo class

  #60 opened a year ago by Yeojoon
  2

• a bug found in save_model of LOMOTrainer

  #54 opened a year ago by DingQiang2018
  10

• Runtime error on 2nd epoch, trying lora only

  #56 opened a year ago by wasifferoze
  0

• hook函数之所以再需要额外操作一次，那里的描述我觉得有些不妥，也或者我理解有问题

  #57 opened a year ago by LaosGAmin
  1

• Customized loss value

  #52 opened a year ago by ZN1010
  4

• LlaMA-7B + LoRA在16GB的V100上OOM

  #53 opened a year ago by zhenqincn
  2

• 选取bloom-1b7作为model和wic作为数据集出现IndexError: tuple index out of range

  #49 opened a year ago by Tonystark64
  1

• 请问自定义Dataset只能是classification的数据集吗

  #51 opened a year ago by Tonystark64
  1

• 小学习率问题

  #50 opened a year ago by wpcool
  1

• Is LOMO a concurrent work of the official implementation?

  #43 opened a year ago by DesperateExplorer
  4

• LOMO是否支持bfloat16模型的训练？

  #46 opened a year ago by Wangyupei
  6

• CLIP梯度和梯度overflow的影响

  #48 opened a year ago by tzjtatata
  2

• Memory Usage continues to grow

  #45 opened a year ago by Jetcodery
  5

• batch size开2后一直提示gradient overflow。。

  #42 opened a year ago by 00drdelius
  16

• Functions to measure the memory usage

  #38 opened a year ago by JiaxiangRen
  1

• 请教个问题，LLM 训练会存在 micro-batch 之间需要累积梯度的场景，这种场景也会有优化吗？

  #34 opened a year ago by nullnonenilNULL
  1

• 公式4疑问

  #37 opened a year ago by yaorong1996
  1

• 关于代码理解和显存占用的问题

  #41 opened a year ago by anbyaa
  1

• llama-33B/llama-65B均报OOM，8*V100跑不起来怎么回事呢？

  #28 opened a year ago by alisyzhu
  7

• 关于微调llama-65b的疑问

  #40 opened a year ago by Facico
  2

• Performance Model after Full Fine-tuning by LOMOTrainer

  #25 opened a year ago by dat-browny
  9

• How to calculate the used GPU memory for each part as in the paper?

  #36 opened a year ago by liming-ai
  2

• LORA+LOMO distributed learning

  #33 opened a year ago by JiaxiangRen
  2

• LOMO+QLoRA简单更改后的报错

  #35 opened a year ago by 00drdelius
  7

• Key Error: LOCAL_RANK

  #31 opened a year ago by snykral
  1

• about torch.stack(self.grad_norms)

  #30 opened a year ago by jinzitian
  3

• 我使用了Resnet50+LOMO优化器，使用cpu去跑，系统内存相比sgd 没有任何变化，请问合理吗

  #29 opened a year ago by yaocy
  0

• Memory consumption first grows up then falls down.

  #26 opened a year ago by zhenqin96
  3

• Some confusion about the method of the paper

  #27 opened a year ago by JorunoJobana
  3