hsiehjackson/RULER
This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?
PythonApache-2.0
Issues
- 3
Raw scores?
#10 opened by WesleyYue - 4
- 2
- 2
- 2
About InterLM2.5
#47 opened by showgood163 - 1
- 2
hope add qwen2-7b-chat result
#46 opened by Chandler-Bing - 0
Gemini flash 1.5 results
#43 opened by augusto-rehfeldt - 2
gpt-4o results?
#12 opened by the21st - 1
RULER with Mamba
#41 opened by Andron00e - 1
- 0
The one-shot example of CWE task
#38 opened by guanzhchen - 1
- 1
questions about ICL code for variable tracking
#27 opened by vkaul11 - 1
- 1
- 1
What is the need for is_icl parameter?
#25 opened by vkaul11 - 1
lost in the middle problem
#24 opened by vkaul11 - 4
prediction evaluation statistics
#22 opened by vkaul11 - 2
- 2
No Generated Output and JSON Serialization Error when calling llm directly in VLLMClient
#11 opened by yaswanth-iitkgp - 1
128K sequence length means 131072 or 128000
#34 opened by syp1997 - 1
Error in hugging face links in README
#35 opened by etienneasln - 3
- 1
Qwen2 and DeepSeek-V2 results?
#33 opened by hijkzzz - 1
pre_sample in qa code
#29 opened by vkaul11 - 3
Base vs Chat prompt question.
#31 opened by karansaxena - 2
request for evaluating GLM4-9B-chat(-1M)
#28 opened by yucc-leon - 5
Prediction format during evals
#30 opened by karansaxena - 10
- 3
what was the reason to use nltk in NIAK task here
#19 opened by vkaul11 - 2
dataset argument for qa.py not specified
#18 opened by vkaul11 - 1
Why do you use partial match max metric for QA
#15 opened by vkaul11 - 0
Question about files nouns.list and verbs.list
#16 opened by vkaul11 - 2
Tempate for Yi?
#13 opened by liyucheng09 - 1
Do we have any ideia how many tokens is used to run the full benchmark in a model?
#9 opened by daniellefranca96 - 4
Llama 3 rope theta
#6 opened by ganler - 2
Time taken on 8 A100?
#7 opened by jzhang38 - 1
Why is multi_key_2 and 3 with only 1 key?
#8 opened by jzhang38 - 4
niah.py hang with hf models
#5 opened by hijkzzz - 0
- 1
- 2
Show Gemini Pro results
#2 opened by s-macke - 3
When will the codes be release
#1 opened by Mooler0410