WIP: This repository is under development.
Unofficial PyTorch/🤗Transformers(+Gemma) implementation of Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
- Paper Link: https://arxiv.org/abs/2404.07143
git clone https://github.com/Beomi/InfiniTransformer
We need to install the latest version(
b109257f4f
) of 🤗Transformers from the source code.
pip install -r requirements.txt
pip install -e git+https://github.com/huggingface/transformers.git@b109257f4f#egg=transformers
rm ./src/transformers/src/transformers/models/gemma/modeling_gemma.py
ln -s $(pwd)/modeling_gemma.py ./src/transformers/src/transformers/models/gemma/modeling_gemma.py
python test.py