Question about the human eval result, thank you.
Opened this issue · 2 comments
guotong1988 commented
In 12 line of human eval result
the human result is 4,750
.
But in the test.jsonl
, the SQL is:
{"phase": 2, "table_id": "2-15062099-1",
"question": "What is the capacity for the stadium of kadrioru stadium and a current manager of sergei ratnikov?",
"sql": {"sel": 3, "conds": [[2, 0, "kadrioru stadium"], [4, 0, "sergei ratnikov"]], "agg": 5}}
In the test.tables.jsonl
, the table is:
{"header": ["Club", "Location", "Stadium", "Capacity", "Current manager"],
"page_title": "2008 Meistriliiga",
"types": ["text", "text", "text", "real", "text"],
"page_id": 15062099, "id": "2-15062099-1", "section_title": "Clubs and venues",
"rows":
[["Flora", "Tallinn", "A. Le Coq Arena", "9,300", "Pasi Rautiainen"],
["Kalju", "Tallinn", "Hiiu Stadium", "500", "Fredo Getulio"],
["Levadia", "Tallinn", "Kadrioru Stadium", "4,750", "Igor Prins"],
["Maag Tammeka", "Tartu", "Tamme Stadium", "2,000", "Sergei Zamogiln\u00f5i"],
["Sillam\u00e4e Kalev", "Sillam\u00e4e", "Sillam\u00e4e Kalevi Stadium", "2,000", "Vadym Dobizha"],
["Tallinna Kalev", "Tallinn", "Kalevi Keskstaadion", "12,000", "Aavo Sarap"],
["Trans", "Narva", "Kreenholmi Stadium", "3,000", "Aleksei Yagudin"],
["Tulevik", "Viljandi", "Viljandi linnastaadion", "2,500", "Marko Lelov"],
["TVMK", "Tallinn", "Kadrioru Stadium", "4,750", "Sergei Ratnikov"],
["Vaprus", "P\u00e4rnu", "P\u00e4rnu Kalevi Stadium", "1,900", "Ants Kommussaar"]],
"caption": "Clubs and venues"}
How can the SQL execution result be [4.0]
?
guotong1988 commented
@whwang299 Thank you.
guotong1988 commented
In 17 line of human eval result
The SQL is wrong. I understand. The SQL do not correspond to the question.
In the test.jsonl
, the SQL is:
{"phase": 2, "table_id": "2-12821570-2",
"question": "What was the highest number of points scored when there was a draw of 2 and when the amount played was 18?",
"sql": {"sel": 7, "conds": [[3, 0, 2], [2, 2, 18]], "agg": 1}}