Liuhong99/Sophia

Issues

• why did you delete SophiaH?

  #53 opened a month ago by Andron00e
  0

• i think it is similar than rmsprop

  #52 opened a month ago by YooSungHyun
  0

• Suggested New experiments- GPT2-small w/ Sophia on Fineweb-10B data

  #51 opened 3 months ago by sanyalsunny111
  1

• Reason for the discrete distribution

  #49 opened 4 months ago by dgm2
  2

• Does Sophia works with activation-checkpointing?

  #50 opened 4 months ago by ahmdtaha
  1

• What are the values of "bs" for vision?

  #48 opened 7 months ago by codonna9
  3

• I cannot reproduce the small and medium size GPT-2 results.

  #46 opened 10 months ago by dahunshin
  7

• NameError: name 'ddp_rank' is not defined

  #37 opened a year ago by ThuanNaN
  1

• Implementing Sophia-H alternative

  #42 opened a year ago by thegodone
  2

• How to run sophia optimizer with huggingface trainer.

  #32 opened a year ago by Dominic789654
  0

• Training on lit-llama failed to get convergence

  #24 opened a year ago by hx-Tang
  3

• Incorrect "General Usage" code for Updating Hessian and Solution

  #44 opened a year ago by HFarkhari
  1

• About `bs`, `hessian_update`, and extending to tasks beyond CLM

  #27 opened a year ago by KeremTurgutlu
  1

• Sophia-H Implementation in third party

  #38 opened a year ago by robotzheng
  0

• Sophia on jax

  #25 opened a year ago by sglucas
  4

• Can't get good results on smaller models

  #30 opened a year ago by tsalex1992
  1

• Unable to reproduce the GPT 2 small results

  #43 opened a year ago by pmpalang
  1

• Trying to reproduce: AdamW better than SophiaG after tuning

  #31 opened a year ago by adefazio
  11

• Few-shot evaluation code available?

  #45 opened a year ago by sanyalsunny111
  1

• Which is the original repo?

  #36 opened a year ago by SagiPolaczek
  2

• Bug in the per-coordinate clipping?

  #34 opened a year ago by vmarkovtsev
  2

• Sophia with multitensor apply / FusedSophia

  #26 opened a year ago by skyshine102
  1

• which is the original code we should use?

  #28 opened a year ago by brando90
  2

• Please package Sophia as a PyPi Package

  #29 opened a year ago by guilt
  0

• Availability of models?

  #21 opened a year ago by ArthurConmy
  1

• Hessian-vector product vs. Hessian estimator

  #23 opened a year ago by zhouyuan
  1

• RuntimeError: Passing `optimizers` is not allowed if Fairscale, Deepspeed or PyTorch FSDP is enabled

  #17 opened a year ago by lw3259111
  2

• Where is the implementation of the estimators?

  #22 opened a year ago by logprobability
  3

• Does Sophia support multiple GPU nodes?

  #20 opened a year ago by skye-glitch
  1

• Does this work with 16-mixed precision

  #16 opened a year ago by tkella47
  3

• Does not reduced CrammingBERT training time

  #18 opened a year ago by tbaggu
  3

• Is Sophia-G a second-order optimzier?

  #19 opened a year ago by Godforever
  0

• Would you be interested in training a small foundational model with Sophia?

  #15 opened a year ago by Mythli
  3

• Training LLMs such as BERT

  #12 opened a year ago by KongMingxi
  3

• Having trouble replicating the result

  #10 opened a year ago by nalzok
  6

• Minor shape error

  #11 opened a year ago by anruigu
  0

• Trying to backward through the graph a second time

  #1 opened a year ago by snykral
  2

• Ambiguous dependency specification

  #5 opened a year ago by nalzok
  1

• Is it applicable for any loss function?

  #3 opened a year ago by subercui
  2

• Incomplete WandB logging

  #6 opened a year ago by nalzok
  1

• Sophia-H Implementation?

  #7 opened a year ago by nalzok
  2

• Evaluation on other domains

  #2 opened a year ago by francqz31
  1

• Does it support BF16?

  #4 opened a year ago by acostin1
  1