Potentially missing the high way pass
Vincent-Li-9701 opened this issue · 1 comments
Vincent-Li-9701 commented
Hello,
Maybe I missed it, but would you mind pointing out where the high way pass of the gMLP block is in the code? Based on the paper, there is a high way path (addition) between the input and the output. I couldn't find it in the gMLPBlock code.
Thank you
chazzmoney commented
Implemented here: https://github.com/lucidrains/g-mlp-pytorch/blob/main/g_mlp_pytorch/g_mlp_pytorch.py#L39-L45
Used in gMLP: https://github.com/lucidrains/g-mlp-pytorch/blob/main/g_mlp_pytorch/g_mlp_pytorch.py#L233
Used in gMLPVision: https://github.com/lucidrains/g-mlp-pytorch/blob/main/g_mlp_pytorch/g_mlp_pytorch.py#L278