Dataset: AQuA
Closed this issue · 8 comments
Homepage: https://github.com/deepmind/AQuA
Example:
{
"question": "A grocery sells a bag of ice for $1.25, and makes 20% profit. If it sells 500 bags of ice, how much total profit does it make?",
"options": ["A)125", "B)150", "C)225", "D)250", "E)275"],
"rationale": "Profit per bag = 1.25 * 0.20 = 0.25\nTotal profit = 500 * 0.25 = 125\nAnswer is A.",
"correct": "A"
}@nomisto The CoT reference answers with A, B, C etc, but we do not have them explicitly in the dataset anymore. But I guess we can standardize to using A, B, C etc for referring to choices with array index 0, 1, 2 etc. Is this already done consistently across datasets?
Ah, I would remove "Answer : B" completely then, to be consistent with other datasets (No other dataset has the answer in the cot and I would delegate this to prompting then.)
A general observation on the current state of AQuA: The style and formatting of the CoTs and the final answers in the CoTs is quite noisy; there is also a substantial amount of spelling mistakes and ad-hoc abreviations.
(this is a comment on the source dataset, not our work based on it)
Ah, I would remove "Answer : B" completely then, to be consistent with other datasets (No other dataset has the answer in the cot and I would delegate this to prompting then.)
Okay!
As said data is quite noisy, so filtering the answer does not really work in a clean way. I'd use the standardized references as you said.
From personal communication with Simon: The datasets refers to A, B, C etc also in other parts of the CoT stream, not only in the final answer, so it would be difficult to remove these references entirely.
I agree that we should standardize on A, B, C schema for multiple choice, which conveniently means that we don't have to do anything for this dataset :)
Error in Aqua dataset (validation set, id: 0)
"question":"Three birds are flying at a fast rate of 900 kilometers per hour. What is their speed in miles per minute? [1km = 0.6 miles]"
"rationale":"To calculate the equivalent of miles in a kilometer 0.6 kilometers = 1 mile 900 kilometers = (0.6)*900 = 540 miles In 1 hour there are 60 minutes Speed in miles/minutes = 60 * 540 = 32400 Correct answer - A"
"answer":"32400"
Error: Multiplied instead of divided.
Error in Aqua dataset (train set, id: 35).
"Krishan and Nandan jointly started a business. Krishan invested six times as Nandan did and invested his money for double time as compared to Nandan. Nandan earned Rs. 6000. If the gain is proportional to the money invested and the time for which the money is invested then the total gain was?"
Answer is 78000, but I think it is 12 x 6000, thus 72000.