naver/sqlova

Question about the human eval result, thank you.

Opened this issue · 2 comments

In 12 line of human eval result
the human result is 4,750.

But in the test.jsonl, the SQL is:

{"phase": 2, "table_id": "2-15062099-1", 
"question": "What is the capacity for the stadium of kadrioru stadium and a current manager of sergei ratnikov?", 
"sql": {"sel": 3, "conds": [[2, 0, "kadrioru stadium"], [4, 0, "sergei ratnikov"]], "agg": 5}}

In the test.tables.jsonl, the table is:

{"header": ["Club", "Location", "Stadium", "Capacity", "Current manager"], 
"page_title": "2008 Meistriliiga", 
"types": ["text", "text", "text", "real", "text"], 
"page_id": 15062099, "id": "2-15062099-1", "section_title": "Clubs and venues", 
"rows": 
[["Flora", "Tallinn", "A. Le Coq Arena", "9,300", "Pasi Rautiainen"], 
["Kalju", "Tallinn", "Hiiu Stadium", "500", "Fredo Getulio"], 
["Levadia", "Tallinn", "Kadrioru Stadium", "4,750", "Igor Prins"], 
["Maag Tammeka", "Tartu", "Tamme Stadium", "2,000", "Sergei Zamogiln\u00f5i"], 
["Sillam\u00e4e Kalev", "Sillam\u00e4e", "Sillam\u00e4e Kalevi Stadium", "2,000", "Vadym Dobizha"], 
["Tallinna Kalev", "Tallinn", "Kalevi Keskstaadion", "12,000", "Aavo Sarap"], 
["Trans", "Narva", "Kreenholmi Stadium", "3,000", "Aleksei Yagudin"],
 ["Tulevik", "Viljandi", "Viljandi linnastaadion", "2,500", "Marko Lelov"], 
["TVMK", "Tallinn", "Kadrioru Stadium", "4,750", "Sergei Ratnikov"], 
["Vaprus", "P\u00e4rnu", "P\u00e4rnu Kalevi Stadium", "1,900", "Ants Kommussaar"]], 
"caption": "Clubs and venues"}

How can the SQL execution result be [4.0]?

@whwang299 Thank you.

In 17 line of human eval result

The SQL is wrong. I understand. The SQL do not correspond to the question.

In the test.jsonl, the SQL is:

{"phase": 2, "table_id": "2-12821570-2", 
"question": "What was the highest number of points scored when there was a draw of 2 and when the amount played was 18?", 
"sql": {"sel": 7, "conds": [[3, 0, 2], [2, 2, 18]], "agg": 1}}