Could AC-SGD be working with like deepspeed zero2/3?

Question

Could AC-SGD be working with like deepspeed zero2/3?

leiwen83 opened this issue 2 years ago · 1 comments

leiwen83 commented 2 years ago

Hi,

I want to know that whether this compression algo could be working with deepspeed's zero or torch's fsdp?

Thx

Answer 1 · 2023-05-10T07:17:00.000Z

Hi @leiwen83 ，

Briefly, we do not have an implementation to support this yet. On the other hand, these two techniques are orthogonal, you can definitely apply these together since FSDP or ZeRO do not relax any computation.

You are more than welcome to contribute a PR in this open-source project to enable this!

Best wishes,
Binhang