Possible typo in adaptive norm
qiminchen opened this issue · 2 comments
qiminchen commented
Line 351/354
and line 359/362
should be norm2
and norm3
?
Lines 338 to 362 in e823742
csjxchen commented
I think it's a typo too. the layer norm2 norm3 are not utilized in the forword
mattiasegu commented
Hi @qiminchen and @csjxchen, sorry for not replying before! This definitely is a typo :)
If you are working on it, you could simply make a PR to fix it and adjust ckpt weights. I will look into it as soon as I find time.