lucidrains/h-transformer-1d

H-Transformer for Cross-Attention?

Vbansal21 opened this issue · 4 comments

Vbansal21 commented

Is it possible to use this architecture for Cross attention?

lucidrains commented

It's not unfortunately :(

  • 👍1
Vbansal21 commented

What if the the code was altered to make it support cross attention? Would that give any meaningful result?

lucidrains commented

@Vbansal21 it's not possible, since there is no notion of locality between the source and target

  • 👍1
Vbansal21 commented

Okay. Well, then closing this issue.

  • 👍1