TIGER-AI-Lab/MMLU-Pro

Issues

• Add Tencent Hunyuan-Large

  #43 opened 3 days ago
  0

• Add Claude 3.5 Haiku

  #42 opened 3 days ago
  0

• New Model | meta-llama/Llama-3.1-405B-Instruct

  #41 opened 3 days ago
  0

• New Model | mistralai/Mistral-Large-Instruct-2407

  #40 opened 3 days ago
  0

• Which DeepSeek-Coder-V2?

  #39 opened 5 days ago
  1

• Add SmolLM2 1.7B

  #38 opened 5 days ago
  1

• New model | Cohere Aya Expanse

  #37 opened 5 days ago
  1

• New model | Yi - Lightning

  #36 opened 12 days ago
  1

• Add Mistral Small v24.09

  #35 opened 16 days ago
  2

• Add Ministral 3B and 8B

  #34 opened 16 days ago
  1

• What is the Arx-0.3 model?

  #31 opened 21 days ago
  1

• Llama-3.1-nemotron-70b-instruct

  #30 opened 21 days ago
  2

• CUDA error: no kernel image is available for execution on the device

  #29 opened 4 days ago
  0

• regarding leaderboard submission

  #26 opened a month ago
  1

• Add Gemini-1.5-Flash-002 and -Pro-002

  #25 opened 21 days ago
  2

• Paper claims there are 10-choices but the test split has varying number of choices (anywhere from 3 to 10)

  #24 opened 3 days ago
  6

• Suggested minimum context length requirement?

  #23 opened a month ago
  2

• Add Qwen2.5 model family

  #22 opened a month ago
  4

• OpenAI o1-preview and o1-mini

  #21 opened 2 months ago
  3

• where is global_record_file="eval_results/eval_record_collection.csv"？

  #18 opened 2 months ago
  1

• eval_results do not contain the actual answer, right?

  #17 opened 2 months ago
  2

• Questionable questions

  #16 opened 2 months ago
  1

• Why dont use chat template for chat model?

  #15 opened 2 months ago
  1

• Variable length of "options"?

  #14 opened 2 months ago
  1

• Possible to remove spam model result

  #13 opened 3 months ago
  1

• Add Grok-2?

  #12 opened 3 months ago
  1

• Support for standard deviation

  #11 opened 2 months ago
  1

• Request for Llama3.1 8B, 70B and 405B

  #10 opened 3 months ago
  5

• Potential coding errors in `evaluate_from_api.py`

  #9 opened 4 months ago
  1

• Regex pattern in extract_final function.

  #7 opened 4 months ago
  11

• Duplicates in test split

  #6 opened 4 months ago
  1

• Different Setup for Different Models?

  #5 opened 4 months ago
  6

• Add Gemma 2 9B and 27B

  #4 opened 4 months ago
  4

• Chat template for instruct models for local eval

  #1 opened 5 months ago
  1