UKGovernmentBEIS/inspect_ai

Adding a scorer without rerunning existing scorers.

Opened this issue · 2 comments

sohaibimran7 commented

I sometimes want to score a task with an additional scorer but not have to rerun the scorers that have already been run. Is there a way to do that?

jjallaire-aisi commented

There is a actually a score() function you can use for this (but I don't think it's properly documented on the website, we will remedy this!): https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/_eval/score.py

sohaibimran7 commented

Doesn't the score function simply return a new EvalLog with only the newly specified scorers? Could there perhaps be a flag that it accepts which when checked, appends the scorer and its results to the EvalLog?

Also, the score functions can only process a single EvalLog at a time. This is quite a minor thing but it'd be nice if it can score multiple Evallogs simultaneously.

Thanks for the quick response btw!