`to_out` bias

Question

`to_out` bias

JamesDeAntonis opened this issue 4 years ago · 3 comments

JamesDeAntonis commented 4 years ago

Hi again,

Could you make it qkvo_bias instead? We would also like to not have bias for to_out

Thanks!

JamesDeAntonis commented 4 years ago

thanks!

🎉1

Answer 1 · 2021-04-20T19:57:59.000Z

@JamesDeAntonis interesting! usually the outward projection does have bias - can you point me at the implementation where it does not, or is this some in-house architecture?

Answer 2 · 2021-04-20T20:03:24.000Z

@JamesDeAntonis https://github.com/lucidrains/performer-pytorch/releases/tag/1.0.7 attn_out_bias = False for you and your team