The code and data for the paper "MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering", accepted to Findings of ACL 2024.
The dataset can be found in the CSV file MedREQAL.csv
Explanation of columns in the dataset:
-
background: Background section of the systematic review.
-
objective: Objective section of the systematic review.
-
conclusion: Authors' conclusion section of the systematic review, used as the gold answer for generative QA task.
-
question: Generated question (from the systematic review Objective section)
-
verdict: Generated verdict (from the systematic review Authors' conclusion section), one of the three: SUPPORTED, REFUTED, NOT ENOUGH INFORMATION.
-
label: Mapped verdict label: SUPPORTED (0), REFUTED (2), NOT ENOUGH INFORMATION (1).
-
strength: Generated label for the strength of systematic review's findings (from the systematic review Authors' conclusion section), one of the three: LOW, MEDIUM, HIGH.
-
category: Automatically assigned medical category of the question / systematic review.