[BUG] [MoE] Typo in Token Drop policy's default value
Closed this issue · 3 comments
passaglia commented
Describe the bug
In the new MoE Token Drop code, the code and documentation expect drop_policy to be either prob or position, but the current default argument is probs.
Proposed fix
#811
yanring commented
Thanks for the fix, @passaglia! We've already got an internal MR that's been reviewed to fix this issue, so it should be synced to GitHub soon. Thanks again!