In mt_bench_inf.jsonl we release a corrected version of the MT-Bench questions that we use for evaluation. Each entry has the following fields:
question_id: The question numbercategory: Which MT-Bench categoryturn: A list with the turnsreference [optional]: A reference answer
Below, we show a few examples of questions, the original GPT-4 Reference answer, and our corrected answer:
Question:
David has three sisters. Each of them has one brother. How many brothers does David have?
GPT-4 Reference Answer:
David has only one brother.
Our Corrected Answer:
David has no brother. He is the one brother of his three sisters.
Question:
The vertices of a triangle are at points (0, 0), (-1, 1), and (3, 3). What is the area of the triangle?
GPT-4 Reference Answer:
To find the area of the triangle, we can use the formula for the area of a triangle with coordinates A(x1, y1), B(x2, y2), and C(x3, y3):
Area = (1/2) * |(x1(y2 - y3) + x2(y3 - y1) + x3(y1 - y2))|
In this case, the coordinates are A(0, 0), B(-1, 1), and C(3, 3). Plugging these values into the formula, we get:
Area = (1/2) * |(0(1 - 3) + (-1)(3 - 0) + 3(0 - 1))|
Area = (1/2) * |(-0 + 3 - 3)|
Area = (1/2) * |0|
Area = 0
The area of the triangle is 0. This means that the three points are collinear, and there is no triangle formed by these points.
Our Corrected Answer:
Area is 3
Overall, we find nearly 25% of examples in the reasoning, math, and coding categories had incorrect reference solutions or had flawed premises.
We release four processed Physics GRE exams, a common graduate school entrance exam for physics students.
physics_gre_scored.jsonl: Exam GR8677 which we report results on.physics_gre.jsonl: Three aditional Physics GRE exams (GR9277, GR9677, and GR0177). Each entry has the following fields:input: The questiontarget_scores: The correct answer choicehas_image: Whether or not the question has an image. We include only questions without an image in our scoring.
| Model | Percentile |
|---|---|
| Inflection-2.5 maj@8 | 85 |
| Inflection-2.5 maj@32 | 95 |
| GPT-4 maj@8 | 97 |
For the Physics GRE, each correct answer is worth 1 point and each incorrect answer results in a -0.25 reduction. To compute score, we make the following assumption:
Raw_Score = Percentage_Correct - 0.25 * (1 - Percentage_Correct)
where Percentage_Correct is computed purely on questions without images. For simplicity, we do not use heuristics to allow the model not to answer.
| Raw Score | Percentile |
|---|---|
| 81 – 100 | 98 |
| 77 – 80 | 97 |
| 75 – 76 | 96 |
| 72 – 74 | 95 |
| 71 | 94 |
| 69 – 70 | 93 |
| 67 – 68 | 92 |
| 65 – 66 | 91 |
| 64 | 90 |
| 63 | 89 |
| 61 – 62 | 87 |
| 60 | 86 |
| 59 | 85 |
| 57 – 58 | 84 |
| 56 | 82 |
| 55 | 80 |
| 53 – 54 | 78 |
| 52 | 77 |
| 51 | 75 |
| 49 – 50 | 72 |
| 48 | 70 |
| 47 | 69 |
| 45 – 46 | 66 |
| 44 | 64 |
| 43 | 62 |
| 41 – 42 | 59 |
| 40 | 57 |
| 39 | 54 |
| 37 – 38 | 52 |
| 36 | 48 |
| 35 | 46 |
| 33 – 34 | 43 |
| 32 | 41 |
| 30 – 31 | 38 |
| 29 | 35 |
| 28 | 32 |
| 26 – 27 | 30 |
| 25 | 27 |
| 24 | 25 |
| 22 – 23 | 22 |
| 21 | 20 |
| 20 | 18 |
| 18 – 19 | 16 |
| 17 | 14 |
| 16 | 12 |
| 14 – 15 | 10 |
| 13 | 9 |
| 12 | 8 |
| 10 – 11 | 6 |
| 9 | 5 |
| 8 | 4 |
| 6 – 7 | 3 |
| 5 | 2 |
| 1 – 4 | 1 |
| 0 | 0 |