Can you please explain how to interpret the output on the main/examples/basic.ipynb
jacobweiss2305 opened this issue · 5 comments
Can you please explain how to interpret the metrics for the last cell on main/examples/basic.ipynb?
https://github.com/microsoft/promptbench/blob/main/examples/basic.ipynb
What is the .947 mean? Maybe there is a formula you can point us to,?
It looks like a standard accuracy metric. How many the bot answered correctly divide by total responses?
The data is pre-labelled so thats my best guess.
What is the .947 mean? Maybe there is a formula you can point us to,?
That looks like an accuracy score
It looks like a standard accuracy metric. How many the bot answered correctly divide by total responses?
The data is pre-labelled so thats my best guess.
Yes reviewing the code you showed it seems to be the prediction score with an accuracy of 94% on each section it is evaluating.
Hi, Millz98 is right, 0.947 means accuracy of this prompt.