This multiple choice question dataset on a fictional organ, the Glianorex, is used to assess the capabilities of models to answer questions on knowledge they have never encountered.
The data generation pipeline is provided in this repository. It supports OpenAI and Anthropic models.
While we provide a pre-generated dataset, we encourage researchers and model evaluators to generate their own private data, ensuring your models have not been contaminated by the dataset and content found in this repository.
Paper: Multiple Choice Questions and Large Languages Models: A Case Study with Fictional Medical Data
The dataset is composed of 976 questions with 4 options, only 1 option is correct.
Questions can be in English or French and are tagged by the language column.
In addition, a generator column contains the name of the model used to generate the data.
- Run
cargo build - Set your OpenAI API key,
export OPENAI_API_KEY=sk-...or Anthropic API keyexport ANTHROPIC_API_KEY=... - Run
cargo run <model>withmodelbeing one ofgpt4,gpt35orclaude35sonnet.
Multiple .json files will be generated containing the different data generated.
book-toc.jsoncontains the table of content of the book.book-content-en.jsonis the entire book in English, respecting the same structure asbook-toc.jsonbook-content-fr.jsonis the entire book in French, respecting the same structure asbook-toc.jsonbook-qa-en.jsoncontains the question and answer samples generated in English based on chapters inbook-content-en.jsonbook-qa-fr.jsoncontains the question and answer samples translated from English