This repo includes codes for the paper PermuteFormer: Efficient Relative Position Encoding for Long Sequences. Directory long_range_arena is for experiments on the dataset Long-Range Arena. Directory language_model is for experiments on the dataset WikiText-103.