[Feature Request]: Standardize All Prompt-based Techniques
Closed this issue · 0 comments
alckasoc commented
Feature Description
All the prompt-based agent methods should be standardized:
k=5
fewshot examples (later we can also makek=1
fewshot examples)- across all methods for each benchmark, use the same
k
-shot examples - each benchmark should have that benchmark's specific fewshot examples
- you will need to, both truncate/pad the fewshot examples to
k
and also ensure thek
fewshot examples for a benchmark is consistent across all methods (at least methods that already have fewshot examples for them)
Reason
No response