twistedcubic/attention-rank-collapse

[ICML 2021 Oral] We show pure attention suffers rank collapse, and how different mechanisms combat it.

Jupyter NotebookApache-2.0