LayerNorm improvements
Closed this issue · 1 comments
FL33TW00D commented
https://fleetwood.dev/posts/layernorm-as-fast-as-possible
Your layernorm is subject to precision errors! Also you're missing eps
.
cryscan commented
Thanks! One of my friends pointed out that I should use Welford's algorithm earlier in development but I was too busy then. Now that I have time, I will do a quick fix.