/transformer-ls

Official implementation of Long-Short Transformer in PyTorch.

Primary LanguagePythonMIT LicenseMIT

Long-Short Transformer (Transformer-LS)

This repository hosts the code and models for the paper:

Long-Short Transformer: Efficient Transformers for Language and Vision

Updates

Architecture

plot Long-short Transformer substitutes the full self attention of the original Transformer models with an efficient attention that considers both long-range and short-term correlations. Each query attends to tokens from the segment-wise sliding window to capture short-term correlations, and the dynamically projected features to capture long-range correlations. To align the norms of the original and projected feature vectors and improve the efficacy of the aggregation, we normalize the original and project feature vectors with two sets of Layer Normalizations.

Tasks