jiaweizzhao/GaLore

Issues

• IndexError: tuple index out of range

  #47 opened 6 months ago by zyushun
  11

• the problem of warmup step and num training step

  #62 opened 3 months ago by BIGKnight
  0

• Questions about reproducing the result of "Benchmark 2: Fine-Tuning RoBERTa on GLUE tasks"

  #44 opened 7 months ago by JamesSand
  2

• loss figure data

  #61 opened 3 months ago by BaohaoLiao
  0

• Zero Loss: The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated singular values

  #58 opened 4 months ago by akjindal53244
  1

• (Question) About glue tasks

  #52 opened 5 months ago by ZhichaoWang091732
  3

• ValueError: can't optimize a non-leaf Tensor (param.is_leaf=False,param.retains_grad=False)

  #60 opened 3 months ago by liveck
  0

• Results vs FP32

  #59 opened 4 months ago by tsengalb99
  0

• Figure 1 clarification on batch size and sequence length

  #57 opened 4 months ago by psandovalsegura
  1

• Questions about glue task report scores

  #56 opened 5 months ago by MYT677
  0

• Support for DDP with multi-gpus

  #55 opened 5 months ago by seongjunyun
  0

• Does galore save gradient memory?

  #53 opened 5 months ago by jinqixiao
  1

• Why not reproject the internal Adam states during update_proj_gap?

  #54 opened 5 months ago by liuliu
  2

• When I used galore on orpo, the learning rate was set to 8e-6, but the training rate was 0.01

  #46 opened 7 months ago by Minami-su
  1

• Galore finetuning #stopped

  #51 opened 6 months ago by j-datta
  0

• How many GB memory is required to train the 7b model using DDP mode with galore?

  #40 opened 7 months ago by zhangqijun
  1

• Memory issue

  #49 opened 6 months ago by fakerybakery
  2

• Hyperparameters for SFT?

  #15 opened 8 months ago by peterjc123
  4

• `torch_run.py` lacking autocast and scaling for Automatic Mixed Precision

  #45 opened 7 months ago by bhavnicksm
  1

• Galore unstable on Llama 7B beyond 20K steps

  #43 opened 7 months ago by kyleliang919
  1

• Questions about Figure 3 in the original paper

  #42 opened 7 months ago by fy817
  0

• ValueError: some parameters appear in more than one parameter group

  #41 opened 7 months ago by jiaohuix
  0

• can support llava model ?

  #39 opened 7 months ago by awzhgw
  0

• Dataset loading issue, integration with Colossal-AI

  #33 opened 8 months ago by Edenzzzz
  3

• Release of Trained Models

  #38 opened 8 months ago by JLake310
  0

• Where is LOMO (fused gradient update) implemented?

  #37 opened 8 months ago by gaotianyu1350
  1

• linalg.svd: The algorithm failed to converge

  #26 opened 8 months ago by Blueman2
  3

• Any plan for the first stable release?

  #36 opened 8 months ago by wsp317
  0

• Third-party benchmark

  #6 opened 9 months ago by hiyouga
  15

• Support for Jamba (ai21labs/Jamba-v0.1)

  #34 opened 8 months ago by creatorrr
  1

• Resume function for optimizer

  #35 opened 8 months ago by bokyeong1015
  0

• Double approximation of second moment in Adafactor

  #8 opened 9 months ago by threewayhandshake
  2

• Please add Phi-2 Support

  #19 opened 8 months ago by calebmor460
  1

• Can't reproduce the result of "Benchmark 2: Fine-Tuning RoBERTa on GLUE tasks"

  #25 opened 8 months ago by CrazyElements
  7

• How to get optim_target_modules=["attn", "mlp"] for other model?

  #27 opened 8 months ago by imrankh46
  4

• A few questions regarding the results and methodology.

  #28 opened 8 months ago by roymiles
  1

• Reproducing Perplexity evaluation

  #30 opened 8 months ago by NitzanHod
  2

• RuntimeError: diag(): Supports 1D or 2D tensors. Got 3D

  #17 opened 8 months ago by drimeF0
  0

• Galore + Lora?

  #9 opened 8 months ago by nivibilla
  4

• GaLore in HuggingFace

  #20 opened 8 months ago by IamExperimenting
  12

• layerwise optimizer raises TypeError about slice indices

  #24 opened 8 months ago by winglian
  2

• How can i do continued pre-training using this?

  #21 opened 8 months ago by Aloukik21
  4

• Galore is not supported for Deepseed Zero3

  #23 opened 8 months ago by youganglyu
  1

• Seems not compatible with DeepSpeed

  #12 opened 8 months ago by geniusalert
  1

• Clarifying GLUE Benchmark Accuracy: Validation or Test Set?

  #13 opened 8 months ago by monk1337
  1

• Confusion about the paper

  #14 opened 8 months ago by CrazyElements
  2

• The first optimizer.step() execution cost extremely long time

  #16 opened 8 months ago by xikaluo
  1

• Training Time

  #3 opened 9 months ago by thisisisheanesu
  2

• CUDA out of memory in torch.linalg.svd

  #4 opened 9 months ago by threewayhandshake
  0

• RuntimeError: cusolver error: CUSOLVER_STATUS_INVALID_VALUE in torch.linalg.svd

  #7 opened 9 months ago by samuelwheeler
  1