hsiehjackson/RULER

RULER with Mamba

Andron00e opened this issue · 1 comments

Andron00e commented

Hi!
Are there any results available for State Spaces?

hsiehjackson commented

Hi! We have results for Jamba (hybrid base model with mamba and transformer) in our paper. We also evaluated this mamba model https://huggingface.co/state-spaces/mamba-2.8b-slimpj which doesn't train with long sequence length. If you have interests in models related to Mamba, I recommend you read this paper https://arxiv.org/pdf/2406.07887. They have results using RULER!