Allow dynamically created tasks in llm_as_judge
Opened this issue · 0 comments
lilacheden commented
Currently the task attribute of LLMAsJudgeBase is a string that accepts either the task suffix (for LLMAsJudge) or the task full path (for TaskBasedLLMAsJudge).
Need to change its type to Task and initialize with either the full task path or a Task object, as done with the template attribute of the same class.
This requires a small but non backward compatible change in LLMAsJudge.