/rerope

Rectified Rotary Position Embeddings

Primary LanguagePython

Rectified Rotary Position Embeddings (ReRoPE)

Using ReRoPE, we can more effectively extend the context length of LLM without the need for fine-tuning.