huggingface/open-muse

Prenorm to Microsoft's ResiDual

isamu-isozaki opened this issue · 1 comments

isamu-isozaki commented

image

Introduced to me by Gothos on discord. The basic idea is this gives a better result than both pre norms and post norms by residually combining them like so

idea

isamu-isozaki commented

closed due to we are already doing something similar with normformer