ofirpress/attention_with_linear_biases

Integration with `transformers`

sayakpaul opened this issue · 1 comments

sayakpaul commented

Amazing work! I'm sure it will open up doors for researchers to think about ways to better extrapolate during inference time.

I am interested to know if you know of any integrations that use AliBi with transformers from Hugging Face.

ofirpress commented

Thanks!

I'm not aware of any correct integrations with the transformers library from HF, but when the BigScience large language model will complete training, it will be available through huggingface and since that model uses ALiBi that will require the integration of ALiBi into the transformers library.

You can read more about it here: https://bigscience.notion.site/BigScience-176B-Model-ad073ca07cdf479398d5f95d88e218c4

  • 🎉1