openai/evals

Should random collection of values be supported?

assert6 opened this issue · 0 comments

if self.num_few_shot > 0:
assert is_chat_prompt(sample["input"]), "few shot requires chat prompt"
prompt = sample["input"][:-1]
for s in self.few_shot[: self.num_few_shot]:
prompt += s["sample"]
prompt += sample["input"][-1:]

refer to evals/registry/evals/cissp-study-questions.yaml

few_shot_jsonl has 143 pieces of data, but num_few_shot only takes the first 4 items.