Liuhong99/Sophia

Does it support BF16?

Closed this issue a year ago · 1 comments

acostin1 commented a year ago

Hi,
Wondering if you have optimized for the BF16 data type like AnyPrecision Optimizer

https://pytorch.org/blog/efficient-large-scale-training-with-pytorch/

https://github.com/pytorch/torchdistx/blob/9c1b9f5cb2fa36bfb8b70ec07c40ed42a33cc87a/src/python/torchdistx/optimizers/anyprecision_optimizer.py#L19

Liuhong99 commented a year ago

yes, bf16 is well supported. we run all experiments with bf16.

👍1