- Before running the python files, make sure you change the line
openai.api_key = "add-your-api-key-here"
to your OpenAI key. - To run either of the python files, use
python file.py input_csv_arg output_csv_arg
where arg is the csv file of the TruthfulQA dataset
- For a list of available models, refer to
openai_models.json
- A temperature of 0 is used and max tokens of 50 is set. All other parameters are default values.
- For model retried answer for chat models, the max token is increased to 60 as the model deterministically replies "I apologize for the confusion".
- Use
completions_nonchat.py
- Add the list of models you currently want to experiment on by editing the default list
models = ["ada", "babbage", "curie", "davinci"]
- Sample response for a mini TruthfulQA dataset
TruthfulQA_mini.csv
(has 10 questions) can be found innonchat_completion_outputs.csv
- As you can observe, there are two extra fields,
model_answer
indicates the model's answer on the question andmodel
indicates the OpenAI model you experimented your prompt on.
- Use
completions_chat.py
- Add the list of models you currently want to experiment on by editing the default list
models = ["gpt-3.5-turbo"]
- Sample response for a mini TruthfulQA dataset
TruthfulQA_mini.csv
(has 10 questions) can be found inchat_completion_outputs.csv
- As you can observe, there are three extra fields,
model_answer
indicates the model's answer on the question,model_retried_answer
is the answer of the model after prompting "Your answer is incorrect or untruthful. Can you retry answering?" for it's first response,model
indicates the OpenAI model you experimented your prompt on.