bitsandbytes-foundation/bitsandbytes

Any changes to the function "optim.AdamW8bit"?

arceus-jia opened this issue · 4 comments

arceus-jia commented

When I use version 0.37.0 of AdamW8bit as an optimizer, the loss calculation quickly becomes nan, (which is problematic when training dreambooth or Lora) but when I fall back to version 0.35.4, these problems will no longer exist

ArrowM commented

Bump. (plus I think this is a dup of #121). Something changed between 0.35.0 and 0.36.0.

ArrowM commented

This is a total shot in the dark, but I wonder if bitsandbytes/functional.py:218-223 was accidentally indented in 2f2063b

rationalism commented

@ArrowM @arceus-jia This should be fixed by 0.41.1

github-actions commented

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.